dataverse

  • poikilotherm (CEST,UTC+2)
    *
    K
    Roy Pardee (KaPoW)
    : to extend a bit on this: making data FAIR does not require public availability of data. Metadata publications are pretty common and one of the usual use cases of Dataverse. It usually is called a "registry" in these cases instead fof "repository". In our installation, we try to focus on the metadata and keep the data out - mostly not because of paranoia but because our institutes very often do not get storing the data longterm wrong and we see not much use in duplicating the data. Also, we have a lot of data that is rather large (TB/PB) and that is simply impossible to have a copy of. In these cases, metadata publications make up a pretty good alternative, as the data is at least FAIR and referencable.
  • *
    K
    Roy Pardee (KaPoW)
    : to extend a bit on this: making data FAIR does not require public availability of data. Metadata publications are pretty common and one of the usual use cases of Dataverse. It usually is called a "registry" in these cases instead fof "repository". In our installation, we try to focus on the metadata and keep the data out - mostly not because of paranoia but because our institutes very often do not get storing the data longterm wrong and we see not much use in duplicating the data. Also, we have a lot of data that is rather large (TB/PB) and that is simply impossible to have a copy of. In these cases, metadata publications make up a pretty good alternative, as the data is at least FAIR and referencable.
  • (We actually did take this one step further and added a new field type to allow real URLs. This is a fork... https://data.fz-juelich.de/guide/juelich/data-linking.html)
  • prsridha
    P
    image.png
  • Thanks for looking into it! This is how it's looking from my end. I have deleted the files manually for now.
  • pdurbin
    P
    prsridha
    : manually? In smaller batches? One at a time? Using the API? Sorry, I'm not sure what you mean. 😅 Thanks for the screenshot! Is that from the demo server? Or your own installation?
  • Pallinger Péter
    P
    Hello! One of our test servers fails to run the dataverse installer (I tried to install v5.13), due to
    "Internal Exception: java.sql.SQLException: Error in allocating a connection. Cause: Access denied to execute this method : setURL"
  • I am a bit stumped to how to debug this issue. I could just reformat the whole VM, but I would like to know why this problem occurs...
  • It would seem that the sql password is bad, but it is the same as on out other test server, which is working fine...
  • First, I would like to know how the POSTGRES_PASSWORD in the default.config is used... How can I check whether it is set correctly?
  • * First, I would like to know how the POSTGRES_PASSWORD in the default.config is used... How can I check whether it is set correctly in payara?
  • Pallinger Péter
    P
    Or could the cause possibly be that setURL is an sql function that is not defined in the current database?
  • pdurbin

    I only see setURL once in the codebase and it's not a SQL thing:

    $ ack setURL src
    src/main/java/edu/harvard/iq/dataverse/DatasetVersion.java
    1609:    public String getReturnToDatasetURL(String serverName, Dataset dset) {
    
  • Pallinger Péter
    P
    I think it is in eclipselink, org.eclipse.persistence. ....
  • * I think it is in eclipselink, org.eclipse.persistence. .... I dug somewhat into the source code after examining the stack trace...
  • I would seem that I somehow misset the SQL connection parameters, but I am still not sure what parameters the server uses. The postgres server itself does not report any connection attempts in the log, although I dialed up the log level to DEBUG1...
  • Which would point to the server adress to be misset, but it seems correct in the domain.xml (is that the correct place for sql connection params like dataverse.db.host, not?).
  • * Which would point to the server adress to be misset, but it seems correct in the domain.xml (is that the correct place for sql connection params like dataverse.db.host?).
  • I can manually connect to postgres with psql, giving the apparently same parameters that dataverse is set to use...
  • pdurbin
    Ok. So you're saying if I use the wrong password for postgres I should expect this error?
  • Pallinger Péter
    P
    maybe, I will try to set the wrong password on my other test server...
  • Pallinger Péter
    P
    Unfortunately, the password error is different:
    FATAL: password authentication failed
    for user "dvnuser"
  • so, I am again stumped about the cause of the above setURL error...
  • pdurbin
    Any ideas,
    O
    poikilotherm (CEST,UTC+2)
    ?
  • poikilotherm (CEST,UTC+2)
    setURL is coming from the underlying JDBC class and refers to some wrong JDBC URL
  • Please check your credentials
  • AFAIK the installer is battle tested to create the aliases etc
  • But of course something might have gone wrong there
  • pdurbin
    containerization working group meeting in 5 minutes: https://ct.gdcc.io
  • Miguel Silva
    In reply to
    pdurbin

  • containerization working group meeting in 5 minutes: https://ct.gdcc.io
  • As soon I finish prototyping my smart DAQ with swarm capabilities I intend to take a peak at the dataverse source code to propose some improvements in regards to live data acquisition and open science ... Almost there
  • 20230427_194922.jpg
  • Here's the newest smart DAQ with SWARM capabilities
  • Available on my public GitHub profile to anyone use
  • ( Open hardware with CC licensing )
  • @archive:matrix.org joined the room
  • pdurbin
    New logs for this channel! Check 'em out! https://archive.matrix.org/r/dataverse:matrix.org
  • pdurbin
  • poikilotherm (CEST,UTC+2)
    Fancy pants!
  • pdurbin
    I know right
  • pdurbin
    My daughter is stage crew for her 8th grade musical, Finding Nemo Jr., premiering two hours from now! I'm off! Have a great weekend! 🐟️
  • imlostlmao joined the room
  • tosigus joined the room
  • Don Sizemore
    D
    @poikilotherm dataverse.files.directory is only used for temporary files these days, correct? I'm asking because I ran across target/classes/META-INF/microprofile-config.properties:dataverse.files.directory=/tmp/dataverse
  • poikilotherm (CEST,UTC+2)
    No it's not!
  • There is a list in the docs where this stuff is used
  • (I asked myself the same thing when going for Mpconfig)
  • This is a sane default for most installations and development
  • Btw this directory is the default which had been present in the code before as well
  • pdurbin
    a pre-existing condition 🤒
  • Don Sizemore
    D
    @poikilotherm if u hapi i'm hapi, just double-checking
  • pdurbin
    Awesome thread about running Dataverse in containers for API testing: https://dataverse.zulipchat.com/#narrow/stream/377090-python/topic/containers.20for.20API.20testing
  • pdurbin
    Community call in a little under 3 hours, including a demo of Pooch! https://dataverse.org/community-calls
  • pdurbin
  • zbenta
    Z
    Pooch looks promissing, it would be great if we could download all the files, metadata and permissions associated with a said dataset and upload them to a different dataverse instance.
  • zbenta
    Z
    We asked our CERT team to do a blackbox security check of our dataverse installation, they found two vulnerabilities, one regarding the jQuery UI 1.13.0 version (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-31160) and another regarding the absence of Anti-CSRF tokens. Should we create an issue
    P
    pdurbin
    ? The version of dataverse we are using is v5.13.
  • pdurbin
  • Z
    zbenta
    : as for moving datasets from one installation to another, I believe this is now possible using the BagIt format.
  • pdurbin

    This message is being deleted (typo)…

  • zbenta
    Z

    zbenta: yuck. Can you please email our security address? https://guides.dataverse.org/en/5.13/installation/config.html#reporting-security-issues

    I'll do so.

  • pdurbin
    Thanks!
  • zbenta
    Z
    Dataverse also has an issue with the session cookie being sent in cleartext, we tried using a session cookie from a coleague in my browser and I was able to impersonate him.
  • pdurbin
    Z
    zbenta
    : please email security about that too
  • Z
    zbenta
    : also, this channel is logged so it's not the best place to talk about security 😅
  • zbenta
    Z

    Dataverse also has an issue with the session cookie being sent in cleartext, we tried using a session cookie from a coleague in my browser and I was able to impersonate him.

    You can skip this one, it was a misunderstanding, as fot the others, we have sent an email regarding the issues.

  • *

    Dataverse also has an issue with the session cookie being sent in cleartext, we tried using a session cookie from a coleague in my browser and I was able to impersonate him.

    You can skip this one, it was a misunderstanding, as for the others, we have sent an email regarding the issues.

  • pdurbin
    Thanks!
  • Z
    zbenta
    : I can also give you access to https://github.com/IQSS/dataverse-security/issues if you like. Please just let me know.
  • zbenta
    Z
    Sure thing, why not
  • pdurbin
    Z
    zbenta
    : done! Do you see an invite at https://github.com/IQSS ?
  • Santiago Florez joined the room
  • Santiago Florez
    S
    Hello everyone
    I want to change the way to display the metadata citation in the subject field. I need the field to be displayed in URL format. I've been thinking about editing the tsv.
  • pdurbin
    S
    Santiago Florez
    : if you get it working, please open a pull request! ❤️
  • Do you have a screenshot or a mockup of how you would like it to look?
  • Santiago Florez
    S
    Yes, hold on a moment
  • dataverse.png
  • pdurbin
    Interesting. And where does "Chemistry" link to?
  • Santiago Florez
    S
    when I click on it, I search for that metadata in all Dataverses and datasets.
  • pdurbin
    I see
  • Santiago Florez
    S
    i´ve modified the displayFormat in the tsv to "<a href='/dataverse/#{dataverse.alias}?q=""#VALUE""' target=""_blank"" rel=""noopener"">#VALUE</a>"
  • pdurbin
    And maybe the same for author. If you click an author, you see all the datasets by that author?
  • Santiago Florez
    S
    Noup, I just need to change that field
  • And I've tried running solr's update***.sh. I don't know if I need to do something else.
  • pdurbin
    did you reload citation.tsv? with curl?
  • Santiago Florez
    S
    jmmm no
  • I gonna try
  • Santiago Florez
    S
    Once I run the curl command, should I see changes to a new data set? or on all the datasets that I had created
  • pdurbin
    both

You're viewing an archive of events from 2023-05-04. Use a Matrix client to start chatting in this room.

May 2023

  1. Sun
  2. Mon
  3. Tue
  4. Wed
  5. Thu
  6. Fri
  7. Sat
  8. 1
  9. 2
  10. 3
  11. 4
  12. 5
  13. 6
  14. 7
  15. 8
  16. 9
  17. 10
  18. 11
  19. 12
  20. 13
  21. 14
  22. 15
  23. 16
  24. 17
  25. 18
  26. 19
  27. 20
  28. 21
  29. 22
  30. 23
  31. 24
  32. 25
  33. 26
  34. 27
  35. 28
  36. 29
  37. 30
  38. 31

Developer options

Toggles

Backend timing

todo: window.tracingSpansForRequest

Room ID

!AmypvmJtUjBesRrnLM:matrix.org