Skip to content

Character encoding weird effects with letter case sensitivity in ContentType header #150

Closed

Description

@PierrePrimot
  • Framework version: 1.1
  • Implementations: all ?

Let's try to explain the problem with 3 examples :

Raw problem

Problem 1 : "Is this setCharacterEncoding effect valid ?"

// GIVEN a plain request
AwsProxyRequest request = new AwsProxyRequest();
AwsProxyHttpServletRequest httpRequest = new AwsProxyHttpServletRequest(request, null, null);

// WHEN setting the character encoding to UTF-8
httpRequest.setCharacterEncoding("UTF-8");

// THEN I get "; charset=UTF-8", which is not a valid property according to Jersey
String actualCharacterEncoding = httpRequest.getCharacterEncoding();
System.out.println(actualCharacterEncoding); // "UTF-8" -> ok

String contentType = httpRequest.getHeader("Content-Type");
System.out.println(contentType); // /!\ "; charset=UTF-8"

It seems to me that "; charset=UTF-8" is not a valid Content-Type...

Problem 2 : "Double raw media type when forcing the setCharacterEncoding"

// GIVEN a request with "Content-Type" equals to "application/json; charset=utf-8"
AwsProxyRequest request = new AwsProxyRequest();
request.getHeaders().put("Content-Type", "application/json; charset=utf-8");
AwsProxyHttpServletRequest httpRequest = new AwsProxyHttpServletRequest(request, null, null);

// WHEN setting the character encoding to UTF-8
httpRequest.setCharacterEncoding("UTF-8");

// THEN I Get "application/json; application/json; charset=UTF-8", which is not what I expected
String contentType = httpRequest.getHeader("Content-Type");
System.out.println(contentType); // /!\ "application/json; application/json; charset=UTF-8" instead of "application/json; charset=UTF-8"

Problem 3 : "Letter case in content-type header changes the behavior"

// GIVEN a request with "content-type" IN LOWER CASE equals to "application/json; charset=utf-8"
AwsProxyRequest request = new AwsProxyRequest();
String contentTypeHeaderLowerCase = "content-type";
request.getHeaders().put(contentTypeHeaderLowerCase, "application/json");
AwsProxyHttpServletRequest httpRequest = new AwsProxyHttpServletRequest(request, null, null);

// WHEN setting the character encoding to UTF-8
httpRequest.setCharacterEncoding("UTF-8");

// THEN I Get "application/json", which is not what I expected
final String contentTypeFromHeaderLowerCase = httpRequest.getHeader(contentTypeHeaderLowerCase);
System.out.println(contentTypeFromHeaderLowerCase); // /!\ "application/json" instead of "application/json; charset=UTF8"
String actualCharacterEncoding = httpRequest.getCharacterEncoding();
System.out.println(actualCharacterEncoding); // /!\ null instead of "UTF-8"

The same test has a different (and correct) behavior with the header spelled "Content-Type" : "application/json; charset=UTF-8" and actualCharacterEncoding = UTF-8.

Context

One could ask how this problem emerged 😃
I am trying to use a SpringBoot REST application based on Jersey (i.e not a pure Jersey app, and not a Spring MVC app neither). I made a custom ContainerHandler inspired from the SpringBootLambdaContainerHandler to achieve this goal (the latter only deals with the 'dispatcherServlet', and I need to deal with the Jersey one from the Jersey starter).

While testing it locally, I figured out the problem. A Spring filter (OrderedCharacterEncodingFilter) sets the characterEncoding to UTF-8, no matter what the content-type header value is (default behaviour).

if (isForceRequestEncoding() || request.getCharacterEncoding() == null) {
      request.setCharacterEncoding(encoding);
}

Later in the request processing, Jersey tries to extract the MediaType and fails (class : InboundMessageContext)

    public MediaType getMediaType() {
        return singleHeader(HttpHeaders.CONTENT_TYPE, new Function<String, MediaType>() {
            @Override
            public MediaType apply(String input) {
                try {
                    return MediaType.valueOf(input);
                } catch (IllegalArgumentException iae) {
                    throw new ProcessingException(iae);
                }
            }
        }, false);
    }

This leads to the following error when the header value is incorrect :

  • Problem 1 : Error parsing media type '; charset=UTF-8' at org.glassfish.jersey.message.internal.MediaTypeProvider.fromString
  • Problem 2 : Error parsing media type 'application/json; application/json; charset=UTF-8' at org.glassfish.jersey.message.internal.MediaTypeProvider.fromString
  • Problem 3 : Too many "Content-Type" header values: "[application/json; charset=utf-8, application/json; charset=utf-8]" at org.glassfish.jersey.message.internal.InboundMessageContext.singleHeader

I was able to find a case that works locally, by setting the header "Content-Type" with "application/json" (notice the header name case letter and remember it does not work in lower case...).

After deploying the lambda, its execution (via API Gateway) failed with a 400 (problem 3), despite a carefully crafted http request (with "Content-Type"). Ironically, by adding smart logging (i.e System.out(headers) in the handleRequest method), it turned out that from all of the headers sent from my RestClient, only one had been transformed into a lower case form : "content-type" (by the API Gateway??).

Well, no matter if it is appropriate to use the Spring encoding filter and Jersey : the 3 problems detailed above show weird behaviors only by using core aws classes.

What do you think about that ?

Activity

  1. sapessi commented on May 17, 2018

    @sapessi
    Contributor

    Hey @PierrePrimot, I'll look into the setCharacterEncoding method, seems to be where the issue is. For unit tests, I would use the request builder just like we do ourselves: https://github.com/awslabs/aws-serverless-java-container/blob/master/aws-serverless-java-container-spring/src/test/java/com/amazonaws/serverless/proxy/spring/SpringAwsProxyTest.java#L65

  2. PierrePrimot commented on May 18, 2018

    @PierrePrimot
    Author

    Hi @sapessi, thx for the reply and advice. You can find a unit test class for the 3 cases in attachement to this message !
    TestCharsetSetting.txt

  3. added this to the Release 1.2 milestone on Jun 15, 2018
  4. self-assigned this
    on Jun 19, 2018
  5. sapessi commented on Jun 19, 2018

    @sapessi
    Contributor

    hey @PierrePrimot, any chance you can test with the 1.2-SNAPSHOT in the core branch of this repo? Simply download or clone the core branch to your local box then run mvn install. Once that's installed in your local cache you should be able to just change the dependency in your pom to 1.2-SNAPSHOT

  6. sapessi commented on Jun 20, 2018

    @sapessi
    Contributor

    release 1.1.1 is going out to maven central

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions