Use CI_PROJECT_ID instead of name of the project in API calls

[gaborauth]

Hi,

I'm trying to use the plugin in my environment and project structure and I feel like I need either some support or I have an idea for a new feature... :)

So, I have some project with group/subgroup/project structure, for example others/toolsforboox/toolsboox-backend.

I have a minimalistic .releaserc.yml:

branches: ['main']

plugins:
  - "@semantic-release/changelog"
  - "@semantic-release/gitlab"
  - "@semantic-release/git"

The semantic-release detects the URLs via the environment variables:

$ echo $CI_PROJECT_PATH
others/toolsforboox/toolsboox-backend
$ echo $CI_PROJECT_URL
https://my.gitlab.url/others/toolsforboox/toolsboox-backend
$ echo $CI_PROJECT_ID
12

Based on the environment variables, the semantic-release try to act via the Gitlab API.

[7:15:55 AM] [semantic-release] › ✔  Run automated release from branch main on repository https://my.gitlab.url/others/toolsforboox/toolsboox-backend.git
[7:15:56 AM] [semantic-release] › ✔  Allowed to push to the Git repository
[7:15:56 AM] [semantic-release] › ℹ  Start step "verifyConditions" of plugin "@semantic-release/changelog"
[7:15:56 AM] [semantic-release] › ✔  Completed step "verifyConditions" of plugin "@semantic-release/changelog"
[7:15:56 AM] [semantic-release] › ℹ  Start step "verifyConditions" of plugin "@semantic-release/gitlab"
2023-05-26T07:15:56.588Z semantic-release:gitlab apiUrl: 'https://my.gitlab.url/api/v4'
2023-05-26T07:15:56.588Z semantic-release:gitlab repoId: 'others/toolsforboox/toolsboox-backend

But the API call based on the repoId fails:

[7:15:57 AM] [semantic-release] › ✘  EMISSINGREPO The repository others/toolsforboox/toolsboox-backend doesn't exist.

When I tried to retrieve the project based on the repoId, like the plugin, it fails:

$ curl --header "PRIVATE-TOKEN: access-token" https://my.gitlab.url/api/v4/projects/others/toolsforboox/toolsboox-backend
{"error":"404 Not Found"}

But, based on the numeric project ID, it works:

$ curl --header "PRIVATE-TOKEN: access-token" https://my.gitlab.url/api/v4/projects/12
{"id":12,"description":"Tools for Boox backend","name":"toolsboox-backend","name_with_namespace":"Other projects / Tools for Boox / toolsboox-backend","path":"toolsboox-backend","path_with_namespace":"others/toolsforboox/toolsboox-backend"

Did I something wrong with the configuration?

I feel like it would work if the plugin pass the repoId as a number from CI_PROJECT_ID environment variable.

[feefladder]

Having the same issue, any idea how to fix?

[fgreinacher]

Hm, this sounds more like a problem on your end.

Accessing the API via project path works if the path is properly URL-encoded. This is the case for semantic-release, but not in your curl invocation.

I guess you're having authentication issues. Did you follow https://semantic-release.gitbook.io/semantic-release/v/beta/usage/ci-configuration#push-access-to-the-remote-repository?

[gaborauth]

Hm, this sounds more like a problem on your end. Accessing the API via project path works if the path is properly URL-encoded. This is the case for semantic-release, but not in your curl invocation.

I went around the problem and looks like the encoded slash ('/' to '%2F') in the URL before the first query parameter is replaced to %252F by security reason in the firewall we are using and the Gitlab cannot process it:
"GET /api/v4/projects/others%252Ftoolsforboox%252Ftoolsboox-backend HTTP/1.1"

Otherwise, it would look like this:
"GET /api/v4/projects/others%2Ftoolsforboox%2Ftoolsboox-backend HTTP/1.1"

In case of CI_PROJECT_ID works without an issue:
"GET /api/v4/projects/12 HTTP/1.1"

I guess you're having authentication issues.

The authentication is fine, the CI_PROJECT_ID based API access is works.

--

Could it be possible to access the Gitlab's API based on the project id instead of the project path? Change the settings of the company's firewall is virtually impossible in my end of the issue... :)

Edit: the other option would be, to have Gitlab process both %2F and %252F encodings. I try to raise a ticket about it.

[fgreinacher]

I went around the problem and looks like the encoded slash ('/' to '%2F') in the URL before the first query parameter is replaced to %252F by security reason in the firewall we are using and the Gitlab cannot process it:
"GET /api/v4/projects/others%252Ftoolsforboox%252Ftoolsboox-backend HTTP/1.1"

This sounds like a bug in your Firewall, %252F is the encoded variant of %2F, which is
incorrect.

You can hack around this by overriding CI_PROJECT_PATH with $CI_PROJECT_ID before running semantic-release. I'd prefer to not introduce new configuration options to keep complexity as low as possible.

[gaborauth]

This sounds like a bug in your Firewall, %252F is the encoded variant of %2F, which is incorrect.

It's not a bug, it's looks like a valid security setting, prevents some of double encoding attacks, but it is not my cup of tea, I cannot change this behavior.

You can hack around this by overriding CI_PROJECT_PATH by CI_PROJECT_ID before running semantic-release.

Okay, I try it, but I think it may have side effects...

I'd prefer to not introduce new configuration options to keep complexity as low as possible.

Okay, last question: what would be the disadvantage if the plugin sends to the Gitlab's API the CI_PROJECT_ID instead of the CI_PROJECT_PATH?

[fgreinacher]

It's not a bug, it's looks like a valid security setting

Oh thanks, I was not aware of this. Not really a standard thing, but still somewhat common it seems: https://stackoverflow.com/questions/5942780/why-is-it-a-security-risk-to-allow-encoded-slashes-in-a-uri

Okay, last question: what would be the disadvantage if the plugin sends to the Gitlab's API the CI_PROJECT_ID instead of the CI_PROJECT_PATH?

Good question! There might be edge cases where folks set CI_PROJECT_PATH explicitly that could then break.

Now it just depends what is more common - folks that have proxies messing with the URL, or folks that set CI_PROJECT_PATH... 🤔

I'd tend to say we can switch to CI_PROJECT_ID but I'd like to hear opinions from other maintainers before proceeding, WDYT @travi @gr2m?

[gr2m]

I don't really know gitlab but your based on the discussion the suggestion sounds good to me 👍🏼

[fgreinacher]

I don't really know gitlab but your based on the discussion the suggestion sounds good to me 👍🏼

Thanks, then let's do it: #576

[travi]

sorry for the delay. returned recently from traveling. i'm also pretty unfamiliar with gitlab, so i support the decision to avoid the problem here.

the one thought that i do want to put out there is that this is a chance to step back to consider if we are doing something wrong at a higher level.

It's not a bug, it's looks like a valid security setting, prevents some of double encoding attacks

i havent given this a ton of thought, but my initial reaction is similar to @fgreinacher's that is a firewall config bug if it is rewriting content in ways that breaks things like this. it would take a bit more evidence to convince me otherwise if we didnt have an alternative like the approach taken to solve this. however, i'm totally open to the evidence supporting that we were doing something wrong with our encoding. it is great that we had an alternative to avoid the issue for this situation, but i'm still concerned about whether that same approach is used elsewhere that will result in other complications. is it worth understanding this a bit deeper to get some insight into whether we have other issues that we should address?

[trickert76]

I'd like to reopen the issue, because the fix above was reverted. I've a similar problem but without any firewall issue.. I'm using a local Gitlab CE release.

Using a `curl -H "PRIVATE-TOKEN: xy" https://my.gitlab/api/v4/projects/ works as expected, when repoId is a numeric number but the url encoded path returns a 404 (in my case it's "sol%2Fdevtools" from "sol/devtools").

My Gitlab instance is behind a nginx reverse proxy and even when the URL doesnt have a query there seems to be some kind of default algorithm to replace %2F back to / (but not %252F, I checked), because the nginx reverse proxy retrieves the %2F, the Gitlab instance behind it retrieves a "/" request for repoId. I'm suprised, that this isn't more often an issue.

I see two options.

a - create an additional property that is either the ID or the PATH and use a default. Replacing the CI_PROJECT_PATH during a CI job run could also be dangerous. It could be by default the PATH, so it's backward compatible.

b - don't call /api/v4/projects/ but the main path /api/v4/projects and parse the returning JSON. This is more work, but it would avoid the wrong message, that the repo doesnt exist.

[fgreinacher]

Sure @trickert76, let's reopen it!

I think an approach like mentioned in #580 (comment) should do the job.