I’ve always put just “BSD” in the package.xml of our projects, mostly because it is the exact value mentioned in https://www.ros.org/reps/rep-0149.html#license-multiple-but-at-least-one . However, this value is not SPDX.
What’s the current best practice? Should we start putting SPDX identifiers into the license tag? Would it deserve a small update of REP 149?
I think it would be ideal to move to SPDX, just because it is explicit.
When I review, I’m not particularly strict on that, but it does seem like a good opportunity for a REP update and some more automation in the GitHub - ros-infrastructure/rosdistro-reviewer: Automation for analyzing changes to the rosdep database and rosdistro index · GitHub
The <license> field in package.xml is used by both rosdoc2 and rosindex to display license information on documentation pages. I tried to understand what they actually use by examining the code. Caveat: did not actually test.
rosindex takes the text from the first of possibly multiple license tags, and displays that string as the “License” in the package summary:
license = REXML::XPath.first(manifest_doc, "/package/license/text()").to_s
rosdoc2 uses catkin_package, which will concatenate strings for possibly multiple tags. rosdoc2 will include the text of a file found in the root directory with the name license.* (ignoring case). I could not find any use of the optional ‘file’ attribute of the license tag.
So “best practice” from the perspective of current implementations of these documentation generators would be to include the string that you actually want displayed as the license summary. Plenty of packages use “Apache License 2.0” as that string for example instead of “Apache-2.0” as recommended by REP 149
I’m not saying any of this is correct or official “best practice”, only what is actually implemented.
Thanks for the details, @rkent . I tried listing rosindex and looking at the licenses, but it seems it’s not possible in the search results page. Only package details show the license. Do you know an easy way to get a listing of all packages with their licenses?
The license field is not included in the database that rosindex uses for search and display, so at the moment there is no way to either display or search for the license field in rosindex tables.
It would not be difficult to add that. The issue though is two:
You could certainly file an issue in rosindex asking to add this. I’d be happy to do the coding, which as I said is fairly easy if you understand rosindex. Whether this is a good idea or not, I’m not sure. As currently managed, it wold largely be a decision that @tfoote would make.
Okay, I went ahead and created the issue: List licenses on the search results page · Issue #650 · ros-infrastructure/rosindex · GitHub .
I also found a very old issue on this topic list of all packages known to ros index with their licenses · Issue #181 · ros-infrastructure/rosindex · GitHub . The issue contains a script that can at least create the CSV with licenses locally.
I ran it for a few distros and here are some trimmed histograms. I’ve put in bold known SPDX identifiers. Apparently, they are there, but the majority does not follow SPDX (including all core ROS 2!).
BSD: 1401
Apache 2.0: 225
MIT: 164
license not found: 88
Apache-2.0: 42
BSD-3-Clause: 27
Apache License 2.0: 27
GPLv3: 26
BSD-3: 25
LGPLv3: 19
LGPL: 19
GPL: 16
Apache 2: 13
Proprietary: 11
GPLv2: 10
BSD 3-Clause: 8
GPL-3.0-or-later: 6
EPL: 6
Apache License 2.0: 764
BSD: 650
Apache-2.0: 191
MIT: 182
Apache 2.0: 169
BSD-3-Clause: 141
GPLv3: 40
Apache 2: 33
Apache License, Version 2.0: 31
license not found: 21
BSD Clause 3: 12
LGPLv3: 11
BSD 3-clause: 10
Apache2: 10
BSD-2-Clause: 8
Apache-2: 7
LGPL: 4
GPL-3.0-or-later: 4
GPL-3.0: 4
GPL: 4
BSD 3-Clause: 4**
MIT License: 3
LGPL-2.1-or-later: 3
GPL-2.0-or-later: 3**
BSD-3: 3
Apache License 2.0: 713
BSD: 601
Apache-2.0: 257
MIT: 218
Apache 2.0: 156
BSD-3-Clause: 148
Apache License, Version 2.0: 34
Apache 2: 31
GPLv3: 30
LGPLv3: 9
BSD-2-Clause: 9
Apache2: 9
license not found: 7
GPL-3.0-only: 7
Apache-2: 7
GPL-2.0-or-later: 5
zlib: 4
LGPL-2.1-or-later: 4
LGPL: 4
GPL: 4
Public Domain: 3
MIT License: 3
GPLv2: 3
GPL-3.0: 3
BSD-3: 3
BSD-2: 3
BSD 3-clause: 3
BSD 3-Clause: 3
ALv2: 3
Apache License 2.0: 585
BSD: 511
Apache-2.0: 161
BSD-3-Clause: 123
MIT: 120
Apache 2.0: 94
Apache License, Version 2.0: 35
GPLv3: 30
license not found: 25
LGPLv3: 9
BSD-2-Clause: 9
Apache2: 9
Apache-2: 7
GPL-3.0-only: 5
GPL-2.0-or-later: 4
GPL: 4
MPL-2.0: 3
MIT License: 3
LGPL-2.1-or-later: 3
LGPL: 3
GPL-3.0: 3
BSD-2: 3
BSD 3-clause: 3
BSD 3-Clause: 3
Apache 2: 3
ALv2: 3
Public Domain: 2
Apache License 2.0: 579
BSD: 479
MIT: 117
BSD-3-Clause: 114
Apache-2.0: 109
Apache 2.0: 95
license not found: 40
GPLv3: 30
Apache License, Version 2.0: 13
LGPLv3: 9
BSD-2-Clause: 9
Apache2: 9
Apache-2: 7
GPL-3.0-only: 5
GPL-2.0-or-later: 4
GPL: 4
MPL-2.0: 3
MIT License: 3
LGPL-2.1-or-later: 3
GPL-3.0: 3
BSD-2: 3
BSD 3-clause: 3
BSD 3-Clause: 3
Apache 2: 3
ALv2: 3
Public Domain: 2
MIT-0: 2
LGPL: 2
GNU Lesser Public License 2.1: 2
CC-BY-3.0: 2
BSL-1.0: 2
I haven’t checked these tables row-by-row, but a quick look says they’re consistent with the exact data from the tables above.
And percentage of total distro packages:
And with some normalization of the license names:
The takeaway for me is that using SPDX strings is okay and wouldn’t break anything. The most common are, however, the non-SPDX license names. Except for “Apache License 2.0” (the text used in OSRA packages), the top 3 most prevalent names are those from REP 149. So I’d say the content of the REP has a great influence (or correlation) on what people put there.
This sounds like a good place to allow which columns are displayed to be dynamically configured.
This sounds painful. Do you know why it’s implemented this way rather than pulling pages from the server as needed?
This would be one true exception where I would be OK with the use of AI to spam repos to update these to the correct strings. Easy bulk task with no harm but saves alot of human time (and scripting time to do the same)
Possible, but an additional complication to the UI. Even ignoring the effort to implement this, I am skeptical that the additional UI complexity is warranted. I’ll probably try an implementation showing the license field, and shrinking some other fields.
I implemented this, so yes I know why it is done this way. I am a generalist hack, that is not a web designer, so I really don’t know if downloading 10MB files in the background is considered foolish behavior or not. The largest files (indexes needed for search) are about 12MB per repo, around 3MB of data after compression. Github web pages has an limit of 25 MB per file, so we have room there. At least on my system (fiber internet, fast laptop) that takes a few hundred milliseconds, usually done in the background, and is cached locally between sessions. So you really have to work hard to see any latency from this. The upside is that search and scrolling is really fast. So it seems to work, and I convinced myself it was not foolish to do this, even though I was initially skeptical. But I am still cautious.
Another side to this is that “pulling pages from the server” IIUC means server-side rendering, for example of search results. That cannot be done using Github pages, so would require a change in the infrastructure beyond a rosindex PR.
That’s a lot less data than I was imagining, and like you thought, probably worth it to get the fast search. I guess the “best” way would be to download piece-meal and cache, but that’s a lot more effort.
