Skip to content

Requests with application/json Content-Type do not correctly assume UTF-8 charset #175

Closed

Description

@eduramiba
  • Framework version: 1.1.1 - 1.1.3
  • Implementations: All

Scenario

It seems that the changes for #150 broke the assumption that all JSON is UTF-8, by definition.

Expected behavior

Parsing a JSON request body to a Java type should assume UTF-8 charset.

Actual behavior

It does not correctly handle characters out of ascii set.

Steps to reproduce

Send request with appplication/json in the Content-Type header and strange characters such as » in the JSON body. It will not be correctly parsed and throw a bad request.

If you set the Content-Type header to application/json; charset=UTF-8 then it works fine, but should not be necessary and most REST clients won't do it.

Full log output

Invalid UTF-8 middle byte 0x65\\n at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 1, column: 13]\\n at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream);

Thanks for your support!

Activity

  1. sapessi commented on Jul 27, 2018

    @sapessi
    Contributor

    Hey @eduramiba, the reason for the change is that HTTP 1.1 says that the default charset is ISO-8859-1. We could automatically detect UTF-8 and adjust the expected charset if one is not specified in the Content-Type header. However, this would break away from the specs and I'm concerned it may affect existing applications that depend on this behavior and are being ported to the framework. I'm hope to suggestions.

  2. eduramiba commented on Jul 27, 2018

    @eduramiba
    Author

    I see, but the thing is that in version 1.1 it works just specifying application/json but no longer does after 1.1.1, and it seems almost all REST clients sush as httpie or even Jersey client only send that. Maybe we could add an specific case just for json, when no charset specified, and default to that?

    See https://stackoverflow.com/questions/9254891/what-does-content-type-application-json-charset-utf-8-really-mean

    http://www.iana.org/assignments/media-types/application/json

    Thanks for the quick response :)

  3. sapessi commented on Jul 27, 2018

    @sapessi
    Contributor

    Perhaps I could add a field for this in the ContainerConfig object, default it to ISO-8859-1, but give you the ability to change this whenever you want:

    LambdaContainerHandler.getContainerConfig()
        .setDefaultContentCharset(StandardCharsets.UTF_8.name())

    Thoughts?

  4. eduramiba commented on Jul 27, 2018

    @eduramiba
    Author

    Yeah I guess that could be a good solution.

  5. eduramiba commented on Aug 27, 2018

    @eduramiba
    Author

    Will this be included in next release?

    Thanks

  6. added this to the Release 1.2 milestone on Aug 27, 2018
  7. sapessi commented on Aug 27, 2018

    @sapessi
    Contributor

    That's the plan.

  8. self-assigned this
    on Oct 22, 2018
  9. sapessi commented on Oct 22, 2018

    @sapessi
    Contributor

    This is now ready in the core branch for release with the next iteration. You can start using it with the 1.2-SNAPSHOT from the branch @eduramiba.

  10. eduramiba commented on Oct 22, 2018

    @eduramiba
    Author

    Thanks! @sapessi

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions