Java
Article

How Optional Breaks the Monad Laws and Why It Matters

By Marcello La Rocca

Java 8 brought us lambdas and streams, both long-awaited-for features. With it came Optional to avoid NullPointerExceptions at the end of stream pipelines that might not return an element. In other languages may-or-may-not-contain-a-value types like Optional are well-behaving monads – but in Java it isn’t. And this matters to everyday developers like us!

Introducing java.util.Optional

Optional<T> is described as “a container object which may or may not contain a non-null value”. That summarizes pretty well what you should expect from it.

It has some useful methods like:

  • of(x), that allows creating an Optional container wrapping on a value x.
  • isPresent(), that returns true if and only if the containPer object does contain a non-null value.

Plus some slightly less useful (or slightly dangerous if you will) ones like get(), that returns the contained value if it’s present, and throws an exception when called on an empty Optional.

There are other methods that behave differently depending on the presence of a value:

  • orElse(v) returns the contained value if there is one, or v by default if the container is empty.
  • ifPresent executes a block of code if and only if there is a value.

Curiously enough, you can see that in its class description there is no mention of methods like map, flatMap, or even filter. All of them can be used to further process the value wrapped by the Optional. (Unless it is empty: Then the functions aren’t called and the Optional stays empty). Their omission might have to do with the fact that in the intentions of the library creators, Optional should not have been a monad.

A Step Back: Monads

Yikes! I can picture some of you sneering when you read that name: monad.

For those who didn’t have the pleasure yet, I’ll try to summarize an introduction to this elusive concept. Be advised and take the following lines with a grain of salt! To go with Douglas Crockford’s definition, monads are “something that once developers really manage to understand, instantly lose the ability to explain to anybody else”.

We can define a monad as:

  • A parameterized type M<T>: in Java terms, public class M<T>.

  • A unit function, which is a factory function to make a monad out of an element: public <T> M<T> unit(T element).

  • A bind operation, a method that takes a monad as well as a function mapping an element to a monad, and returns the result of applying that function to the value wrapped in the monad:

    public static <T, U> M<U> bind(M<T> monad, Function<T, M<U>> f) {
        return f.apply(monad.wrappedValue());
    }
    

Is that all there is to know about monads? Not really, but that is enough for now. Feel free to check the suggested readings at the end of the article if you’d like to read more on the subject.

Is Optional a Monad?

Yes and no. Almost. Definitely maybe.

Optional per se qualifies as a monad, despite some resistence in the Java 8 library team. Let’s see how it fits the 3 properties above:

  • M<T> is Optional<T>.
  • The unit function is Optional.ofNullable.
  • The bind operation is Optional.flatMap.

So it would seem that Optional is indeed a monad, right? Not so fast.

Monad Laws

Any class, to truly be a monad, is required to obey 3 laws:

  1. Left identity, applying the unit function to a value and then binding the resulting monad to function f is the same as calling f on the same value: let f be a function returning a monad, then bind(unit(value), f) === f(value).
  2. Right identity, binding the unit function to a monad doesn’t change the monad: let m be a monadic value (an instance of M<T>), then bind(m, unit) === m.
  3. Associativity, if we have a chain of monadic function applications, it doesn’t matter how they are nested: bind(bind(m, f), g) === bind(m, x -> g(f(x))).

Both left and right identity guarantee that applying a monad to a value will just wrap it: the value won’t change nor monad will be altered. The last law guarantees that monadic composition is associative. All laws together make code more resilient, preventing counter-intuitive program behaviour that depends on how and when you create a monad and how and in which order you compose functions that you will use to map a monad.

Optional and Monad Laws

Now, as you can imagine, the question is: Does Optional<T> have these properties?

Let’s find out by checking property 1, Left Identity:

Function<Integer, Optional<Integer>> f = x -> {
    if (x == null) {
        x = -1;
    } else if (x == 2) {
        x = null;
    } else {
        x = x + 1;
    }
    return Optional.ofNullable(x);
};
// true, Optional[2] === Optional[2]
Optional.of(1).flatMap(f).equals(f.apply(1));
// true, Optional.empty === Optional.empty
Optional.of(2).flatMap(f).equals(f.apply(2));

This works both for empty and non-empty results. What about feeding both sides with null?

// false
Optional.ofNullable((Integer) null).flatMap(f).equals(f.apply(null));

This is somehow unexpected. Let’s see what happens:

// prints "Optional.empty"
System.out.println(Optional.ofNullable((Integer) null).flatMap(f));     
// prints "Optional[-1]"
System.out.println(f.apply(null));

So, all in all, is Optional a monad or not? Strictly speaking it’s not a well-behaving monad, since it doesn’t abide by the monad laws. However, since it does satisfy the definition of a monad, it could be considered one, although one with some buggy methods.

Optional::map and Associativity Law

If you think we got out of luck with flatMap, wait to see what happens with map.

When we are using Optional.map, null is also mapped into Optional.empty. Suppose we map again the result of the first mapping into another function. Then that second function won’t be called at all when the first one returns null. If, instead, we map the initial Optional into the composition of the two functions, the result would be quite different. Check out this example to clarify:

Function<Integer, Integer> f = x -> (x % 2 == 0) ? null : x;
Function<Integer, String > g = y -> y == null ? "no value" : y.toString();

Optional<Integer> opt = Optional.of(2);  // A value that f maps to null - this breaks .map

opt.map(f).map(g);                      // Optional.empty
opt.map(f.andThen(g));      // "no value"

By composing the functions f and g (using the handy Function::andThen) we get a different result than we got when applying them one by one. An even more obvious example is when the first function returns null and the second throws a NullPointerException if the argument is null. Then, the repeated map works fine because the second method is never called but the composition throws the exception.

So, Optional::map breaks the associativity law. This is even worse than flatMap breaking the left identity law (we’ll get back to it in the next section).

orElse to the Rescue?

You might think it could get better if we use orElse. Except it doesn’t.

It is easy to create a chain with more than two functions, where getting null at different stages can lead to different results. Unfortunately we don’t have a way, at the end of the chain, to tell where null was first handled. And so no way to provide the right result when orElse is applied. More in abstract, and even worse, by using orElse we would be relying on the fact that every developer maintaining our code, or every client using our library, will stick to our choices and keep using orElse.

What’s The Catch with Optional?

The problem is that by design non-empty Optionals can’t hold null. You might legitimately object it is designed to get rid of null, after all: And in fact Optional.of(null) will throw a NullPointerException. Of course null values are still common, so ofNullable was introduced to keep us from repeating the same if-null-then-empty-else-of check all over our code. However – and here is the essence of all evil – Optional.ofNullable(null) is translated to Optional.empty.

The net result is that, as shown above, the following two situations can lead to different results:

  • Applying a function before wrapping a value into Optional;
  • Wrapping the value into an Optional first and then mapping it into the same function.

This is as bad as it sounds: it means that the order in which we apply functions matters. When we use map, as we saw, it gets even worse, because we lose associativity invariance as well and even the way functions are composed matters.

In turn, these issues make adding bugs during refactoring not just possible, but even frighteningly easy.

A Real World Example

Now, this might look as an example built ad-hoc to cause trouble. It’s not. Just replace f with Map::get (which returns null when the specified key is not contained in the map or if it is mapped to the value null) and g with any function that is supposed to handle and transform null.

Here is an example closer to real world applications. First, let’s define a few utility classes:

  • Account, modeling a bank account with an ID and a balance;
  • Balance, modeling an (amount, currency) pair;
  • Currency, an enum gathering a few constants for the most common currencies.

You can find the full code for this example on GitHub. To be clear, this is not, by any means, to be intended as a proper design for this classes: We are greatly simplifying, just to make the example cleaner and easy to present.

Now let’s say that a bank is represented as a collection of accounts, stored in a Map, linking account IDs to instances. Let’s also define a few utility functions to retrieve an account’s balance in USD starting from the account’s ID.

Map<Long, Account> bank = new HashMap<>();

Function<Long, Account> findAccount = id -> bank.get(id);
Function<Account, Balance> extractBalance = account -> account != null
        ? account.getBalance()
        : new Balance(0., Currency.DOLLAR);
Function<Balance, Double> toDollars = balance -> {
    if (balance == null) {
            return 0.;
    }
    switch (balance.getCurrency()) {
        case DOLLAR: return balance.getAmount();
        case POUND: return balance.getAmount() * 1.3;
        case EURO: return balance.getAmount() * 1.1;
        default: return 0.;
    }
};

We are now ready to see where individual mapping of our three functions works differently from their composition. Let’s consider a few different cases, where we start from an account’s id wrapped in an Optional, and we map it to the dollar amount for that account.

Optional<Long> accountId3 = Optional.of(3L);
Optional<Long> accountId4 = Optional.of(4L);
Optional<Long> accountId5 = Optional.of(5L);

bank.put(4L, null);
bank.put(5L, new Account(5L, null));

Both for an empty Optional and for a non-empty one wrapping the ID of a non-null account with proper balance, it doesn’t matter whether we map our functions one by one or whether we use their composition, the output is the same. We omit these cases here for brevity, but feel free to check out the repo.

Let’s instead try out the case where an account’s ID is not present in the map:

accountId3.map(findAccount)
        .map(extractBalance)
        .map(toDollars)
        .ifPresent(System.out::println);  // Optional.empty
accountId3.map(findAccount
        .andThen(extractBalance)
        .andThen(toDollars))
        .ifPresent(System.out::println); // 0.0

In this case, findAccount returns null, which is mapped to Optional.empty. This means that when we map our functions individually, extractBalance will never be even called, so the final result will be Optional.empty.

If, on the other hand, we compose findAccount and extractBalance, the latter is called with null as its argument. Since the function “knows” how to handle null values, it produces a non-null output that will be correctly processed by toDollars down the chain.

So here we have two different results depending only on the way in which the same functions, taken in the same order, are applied to our input. Wow!

The same thing happens if we store null in the map for an ID or if the account’s balance is null, since toDollars is similarly crafted to handle null. Check out the repo for further details.

Practical Implications

Besides theoretical disputes on the nature of Optional, there are plenty of practical consequences of the fact that Optional::map and Optional::flatMap break the monad laws. This in turn prevents us from freely applying function composition, as we were supposed to have the same result if we apply two functions one after the other, or their composition directly.

It means that we can no longer refactor our code freely and be sure the result won’t change: Dire consequences might pop up not just in your code base, but – even worse – in your clients’ code. Before restructuring your code, you would need to know if the functions used anywhere in everybody’s code handle null or not, otherwise you might introduce bugs.

handcuffs

Possible Fixes

We have two main alternatives to try to make this right:

  • Don’t use Optional.
  • Live with it.

Let’s see each one of them in details.

Don’t Use Optional

Well, this looks like the ostrich algorithm mentioned by Tanenbaum to solve deadlock. Or ignoring JavaScript because of its flaws.

Some Java developers explicitly argue against Optional, in favour of keeping up with using null. You can certainly do so, if you don’t care about moving to a cleaner, less error-prone style of programming. TL;DR: Optional allows you to handle workflows where some input might or might not be present through monads, which means in a more modular and cleaner way.

Live with It

Optional breaks the monad laws under a couple of circumstances and there’s nothing to be done about it. But we can learn a couple of things from the way we took to come to this conclusion:

  • When starting a chain with a possibly null value and a function that returns an Optional, be aware that applying the function to the value can lead to a different result from first creating the Optional and then flat-mapping the function. This, in turn, can lead to the function not being called, which is a problem if you depend on its site effects.
  • When composing map chains, be aware that, while the individual functions were never called with null, the merged function might produce null as an intermediate result and pass it into non-nullsafe parts of the function, leading to exceptions.
  • When decomposing a single map into several, be aware that while the original function was executed as a whole, parts of it might now end up in functions that are never called if a previous part produced null. This is a problem if you depend on those parts’ site effects.

As a rule of thumb, prefer flatMap over map, as the former abides by the associativity law, while map doesn’t, and breaking associativity is far more error prone than breaking left identity. All in all it is best not to view Optional as a monad that promises easy composability but as a means to avoid having null pop up as the (flat-)mapped functions’ arguments.

I have worked out a few possible approaches to make code more robust, feel free to check them out on GitHub

Conclusions and Further Reading

We’ve seen that certain refactorings across flatMap and map chains can cause the code to change its behavior. The root cause is that Optional was designed to avoid stumbling into NPEs and in order to achieve that it transforms null to empty. This, in turn, means that such chains are not fully executed but end once an intermediary operation produces empty.

It is important to keep this in mind when relying on the site effects the potentially-not-executed functions cause.

If you’d like to read more about some of the topics in this article:

  • Carlos Obregon

    I don’t agree with you. The first thing is that Java’s Optional was never intended to be a Monad and their authors have said it many times. So you can’t blast Optional for not being a Monad if it never was never intended to be one. You might think that it should be a Monad but that’s another discussion. So you are right that Optional breaks the monad laws, but I don’t think that’s a big deal. You can use it for what it was made!

    Your argument on having consequences on refactors is, in my opinion, broken. Consider this class:

    Collection collatz() {
    Collection numbers = new ArrayList();
    numbers.addAll(Arrays.asList(6, 3, 10, 5, 16, 8, 4, 2, 1));
    return numbers;
    }

    Now let’s say I want to “refactor” it, because I want to have the best performance possible, so I try 2 versions:

    Collection collatz() {
    Collection numbers = new TreeSet();
    numbers.addAll(Arrays.asList(6, 3, 10, 5, 16, 8, 4, 2, 1));
    return numbers;
    }

    And

    Collection collatz() {
    Collection numbers = new HashSet();
    numbers.addAll(Arrays.asList(6, 3, 10, 5, 16, 8, 4, 2, 1));
    return numbers;
    }

    But each of the versions will return the numbers in a different order, so I might argue that TreeSet and HashSet are broken! But it’s obvious that they are not broken, they are doing what they were made for! When I’m refactoring I need to ensure that the new code would not change the apparent behavior of the method, if I change it then I’m not refactoring it and it’s my fault.

    So I would say the real conclusion of you article should be “Stop using Optional as a Monad, and use it for what it was made”. And in your fixes you should have say: When using Optional remember that if used in conjunction with legacy code a.k.a. code that return null or poorly written a.k.a. code that accepts a null value and since it’s not a monad, the order in which you compose the functions do matter (the same as “if the order of insertion is important, don’t use TreeSet or HashSet”) And finally if you really think that Optional should be a Monad, you are free to use one of the available options like javaslang’s Option

    • Hi Carlos,
      thanks for your feedback!

      I believe you are missing a couple of points here.

      1) Optional **IS** a monad. Whether they did intend it or not, it doesn’t matter. Check here to see why http://mail.openjdk.java.net/pipermail/lambda-dev/2013-February/008314.html
      (sorry this was supposed to be linked in the article, I must have removed it by mistake)

      2) In your counter example, you are changing one data structure to a different one, one implementation of an interface for another. It’s not only possible, but expected, that the final result is going to be different: it’s a matter of choosing the right data structure for the task.
      When you compose or split functions, instead, there is a reasonable expectation that the result won’t change: that’s because we expect functions composition to be associative.
      Changing the underlying data structure is not just refactoring ;-)
      By definition, in fact, “Code refactoring is the process of restructuring existing computer code—changing the factoring—without changing its external behavior.”
      So comparing the two situations is a bit like apple and oranges: we can’t derive a conclusion on one from the other.

      Summing it up, your advice about being careful with legacy code is a good one, but unfortunately it’s not enough. When you work in teams, often the code you write today will likely be maintained by someone else in a year – when maybe you will be working in a different team or company.
      That’s why we need to try to write code as clean as possible. Leaving this sword of Damocles on your team’s head is not something you should want.

      Using `Option` or another similar third party class is, indeed, an even better advice, of course (unless using 3rd party is not an option).
      But I believe it’s good to know why you need to, and even better it’s having another _option_ (pun intended)

      • Carlos Obregon

        Thanks Marcello for your reply!

        1) Optional IS NOT a Monad. In the documentation there is no mention of being one, and if you read interview or talks of Stuart Marks or Brian Goetz they will tell you that. Java’s Optional is Monad-ish because it has unit, map and flatMap but besides having the correct names for the method, as you pointed in the article, the class doesn’t behave as one. So if it doesn’t walk like a duck, nor swims like a duck, even if it looks like one you must come to the conclusion that it is not a Duck. So Java’s Optional is not a Monad, they just wanted to have a type to represent the possible absence of a value and they put some useful methods that make it look like one, but just incidentally.

        2) I don’t think my example and yours are so different. As you said, changing implementations of an interface should probably change the results. Composing functions using different methods means that the result is probably different, specially when the documentation in Monad clearly states what you should expect after using map and flatMap. So it’s not apple and oranges, is the same: when you do a refactor you need to know what is the behavior of the methods / classes you are using to be sure that the result haven’t changed.

        So I agree that’s important that people don’t expect Optional to be a Monad, but just because it isn’t one doesn’t mean it’s broken. I use it an even if I know that composibility in the strictest sense is broken it let me writes much cleaner code than without it. And as I mentioned before you are always free to use a third-party library if you want a true Monad.

        • Carlos, I’m sorry to have to contradict you, but `Optional` is a monad, even if you wish it wasn’t.
          It doesn’t matter what you, I, Stuart Marks or whoever thinks.
          It matches the definition of a Monad. It might be a ill-behaving one, violating the monad laws, but still a monad (monad laws aren’t included in the definition of monads). If you take the time to read the link above and some of the suggested reading, I’m sure you’ll convince yourself.

          To put it in your terms, you are using the duck-typing in the wrong direction: if a duck can’t fly, does that make it a dog, or just a duck that can’t fly? (the answer is: if that’s all you know, just by the fact that it doesn’t fly, or walk or swim… then there is no way to tell!)

          But the main point is not what Optional is from a theoretical point of view! It’s all about writing bug-free, clean, maintainable code.
          If Optional wasn’t breaking composition, I wouldn’t have cared if they called a monad, a duck or a whatever…
          It’s just that, if you have something that matches the definition of a monad (even if you don’t want to design it as one), and you don’t make sure it also abides by monad laws, bad things happen.

          Now let’s move to your example, and let me show a parallel with arithmetic operations:
          1) Composition: 1 + 2 + 3 == 1 + (2 + 3)
          2) Changing implementation (your example): 1 + 2 + 3 1 * 2 * 3

          We expect associativity-invariance, we definitely don’t expect invariance when changing the operation we run.

          You should indeed keep using Optional, by all means! It does allow for cleaner code, no doubt about that.
          As for everything, there are good practices, and bad practices.
          Putting yourself in a situation that makes creating bugs, and what’s worse silent ones, that you’ll notice only after some time, is not a good practice.
          I’ll bet a beer that someday, when, a few years from now perhaps, you’ll experiment yourself this kind of bug caused by Optional, you’ll have an epiphany and see why this was bad.
          Cheers!

          • Carlos Obregon

            Hi Marcello! Again, thanks for your reply.

            I think that as long as you use an API that you understand. you’ll never have unexpected bugs because of them. I have had my share of bugs that after found I realize where the result of simply don’t understanding how the API really worked and that I use them thinking how I expected to work.

            For sure you sometimes say: I think this is dumb, it’s not how I think it should be. When you create an API you hope that all users will agree that’s useful but that’s not always the case.

            When I read the documentation of Optional it make sense to me. I really enjoy that I can use map/flatMaps without worrying about null. I also understand that Optional is not a monad and the fact that I can’t rely on the third Monad law. But it hasn’t bother me, I just take it into account when I write code with it.

            I also understand that people would expect Optional to comply with the Monad laws, and thankfully in modern times you can almost every time find a 3rd party library to do it.

            So again, thanks for rising the awareness of what Java’s Optional is not!

          • Hi Carlos, thanks for your comments!
            Yes indeed, the goal of the article was not to put shame on Optional – not at all. I hope that’s not the impression it was giving.
            Rather, our goal was to show strengths and weaknesses, and provide patterns that allows anyone to (more or less, or at least a little more) safely use Optional as it is.

            Every design choice is a tradeoff (in this case between allowing `null` as a value and be compliant to monad laws) and I trust this one was made for very good reasons.

            The other side of the spectrum would probably be Scala’s Option class, were `Some(null)` is allowed and not transformed to `None`.

            They are both valid design choice, with PROs and CONs. Indeed, raising awareness about both of them, and issue connected with them, was the second main goal of the article.

            Thanks again for the great feedback!

  • Alejandro Gervasio

    Hi Marcello,

    First and foremost, I’d like to say your article is one of the most thorough ones I’ve seen so far around on the web covering a pretty tangled topic like Monads, specially for those developers (I include myself in this crowd) without a strong background on FP. Having worked for decades with several imperative languages, your post helped me a lot to grab the underlying power of Monads, and particularly how to exploit their functionality with Optional in Java. Leaving the debate aside that Carlos cleverly raised up, do you think that Monads have a place in an imperative language like Java? Is it just another tool to create cleaner and more reliable code? I’d like to hear your opinions.

    Cheers!

    • Hi Alejandro,
      thank you so much for your nice words, I’m glad you enjoyed the article and found it useful.

      That’s a very good question! To which, unfortunately, there is no easy answer…

      First, it depends on what you mean exactly.
      Monads are an abstract concept from category theory. The design of a data structure can acknowledge or ignore it, but the underlying abstract data structure will independently be, or not be, a Monad nonetheless.
      For example, Optional: the abstract data structure that express the absence of a value, or its presence (by wrapping it) it’s the Maybe-monad.
      It is possible to provide an implementation that makes things easy and provides methods like `map` and `flatMap`, or ignore the monad part altogether and strip those methods.
      The only effect of stripping those methods is that the bind operation for the class would be

      “`
      Optional a = …
      Optional b = …
      Function<A, Optional
      f = …
      if (a.isPresent) {
      b = f.apply(a.get());
      } else {
      b = Optional.empty();
      }
      “`

      rather than:

      “`
      Optional
      a = …
      Function<A, Optional f = …
      final Optional
      b = a.flatMap(f);
      “`
      (Example taken from
      http://mail.openjdk.java.net/pipermail/lambda-dev/2013-February/008314.html)

      Then, for an implementation of a monad there is the question about abiding by the monad laws. Complying will have its advantages, but also not allowing `null` has others. It’s a tradeoff, like most decisions.

      But to get back to your question, if you meant it as “there is room to add typical monad methods like map and flatMap”, I’d say that I’d rather have them: they are no harm to who’d rather ignore them, plus you won’t force developers that find them useful to define workarounds as above. And let them write cleaner code.

      If, instead, you meant if there is room in Java to add monad-classes like Optional, I guess in turn we could ask: is there a place for functional programming in Java?

      I believe there are a few aspects to weight:
      1) Resistance from part of the Java community
      2) Integration with Legacy code and frameworks
      3) Integration with the imperative part of the language
      4) Performance and overhead vs possible advantages (clean and robust code, as you mentioned)

      Take streams, for example.
      They are certainly useful, a great way to make short and clean code.
      However:
      1) They come at a cost, https://dzone.com/articles/whats-wrong-java-8-part-iii
      2) Sometimes (even often) using for-each will allow for cleaner, shorter and faster code than converting a collection into a stream then map, filter etc… and finally collect the results back.

      So, is it worth to have FP in Java? Is there room for it?
      Right now, I’d be lying if I’d say I’m sold to yes for the former.
      Java seems very imperative-oriented, and there are alternatives running on the JVM for FP. Forcing all Java developers to go functional, I’d it would probably be a mistake.

      On the other hand, I believe there is room to experiment with some light FP, as it was done.
      I certainly find lambdas, streams, Optional most useful and I’m glad they were added.

      • In the end, trends come and go constantly.
        Recently I was reading an article (that I only partly agree with)
        https://medium.com/@cscalfani/goodbye-object-oriented-programming-a59cda4c0e53#.p2hhkfh36
        advocating dismissal for OO-programming in favour of FP (it’s not the first one, nor the last, of course).
        Before that, procedural programming was the way: I met a worrying number of developers who haven’t yet adapted to OO!
        Everything takes its time, and nothing works for everyone.

        • Alejandro Gervasio

          Hi Marcello,

          Thanks for such a detailed response! I couldn’t ask for anything better, trust me. So, yes my question was if there is room in Java for some nifty FP features, including lambdas, streams and Optional. At first sight, there is, even when many developers with a strong background on OOP are still reluctant to use them in production. In my personal case, I found them quite useful, and I must confess it took me quite a while to make the click and understand what Monads are and how they operate under the hood. But once you get the their rationale, it’s pretty appealing to start working with them.

          So, once again thanks for clarifying all those relevant concepts.

          • Carlos Obregon

            Hi Alejandro, let me give you my two cents in this: I think the most important trait in the code we write is readability. Everything that makes for better readability is more than welcome in any language no matter where they come from! So in the end the question you need to ask is: do I find a code written using FP-constructs such as Optional, more readable than code that doesn’t use them? If your answer is “yes” then by all means use them.

            Bjarne Stroustrup, father of C++, always says that he think is wrong to say that C++ is “only” an OOP language, or “only” an imperative language and so on. I think he’s right about all languages. Languages are not “only” one thing. As languages evolve they bring ideas that would make them still relevant in the the next 30+ years! So Java added Optionals because they are better that nulls as return types. And they added Stream because they are so much better that using a for with an iterable and processing elements. And if they find that using more FP-constructs will help them be more relevant, then I don’t doubt they will add them.

            I think learning these new features is very important because they are added because they will probably make you write better code. But as Marcello says it takes time, so the more important thing is to start using them and try to read, a little, about them so you can learn from the experience of others what works best and adapt it to what you feel is right.

          • Alejandro Gervasio

            Hi Carlos,

            Thanks for joining us in discussing this topic. We’re just a few, but the experience is enriching, that’s for sure. So, I totally agree with you in every point you’ve just raised up. That’s why I’m embracing these features and learning how to use them to produce more readable code.

            Cheers!

          • Thanks Alejandro!
            If you are interested in FP I suggest you try these Coursera specialization about Scala:
            https://www.coursera.org/specializations/scala

            Even if you are not that interested in Scala, it’s good theoretical stuff that you can bring to Java as well.
            I’m not sure if there is still a free “audit only” version of the individual courses, but afaik that should be the case.
            Cheers!

          • Alejandro Gervasio

            Thanks Marcello for posting the link to the Coursera specialization. I”ll check it out in more depth and see what good stuff I can get from it to use it with Java.

            Once again, great post!

  • JuanManuel Gimeno Illa

    After reading your post I’m not sure if the problem is in the Optional or in the functions you use. IMHO functions such as:

    Function<Integer, Optional> f = x -> Optional.ofNullable(
    x == null
    ? -1
    : (x == 2 ? null : x + 1));

    which accept a null as Integer are not “valid functions” when trying to use “mathematical” arguments about monads.

    This function is not Integer -> Integer because an argument of null will be simply unacceptable. So, for me, the problem is not in the Optional but in the function (its a valid function in Java, but not a valid function to reason about monadness).

    Besides, Optional.ofNullable is not unit: unit of the Maybe monad never returns an empty optional. The problem, again, is that in Java a function Integer -> Optional accepts nulls as a valid argument which, from the mathematical point of view, has no sense. The behaviour od Oprional.of is more correct (it simply fails).

    Summing up: the culprit is in the fact that in Java all reference types accept null as a value, not the Optional type.

    • Hi JuanManuel,
      thanks for your comment, that’s an interesting point of view.

      My take is neither is to blame on its own, but when Optional is combined with such functions, bad things can happen.
      So indeed, as part of my final advice, there was a warning not to use such functions – this advice has been moved to the github repo together with a (hopefully) thorough search among the possible solutions.

      A few thoughts/open questiosn (again, not meant to blame Optional):

      1) The same counter-example would work with functions String -> Option, or T -> Option, every time when null is treated as a special case – hence possibly removing the concern about mathematical meaning.
      2) Java allowed null for reference types for long before Optional was designed… Is it it fair to blame the errors on this? I mean, I’m not so sure blaming is retroactive! :D
      3) Scala as well allows null to be assigned to references. The design for Option is different from most Java implementation of the Maybe monad, and it allows null to be contained in a non-empty Option. Not saying it’s better or worse, but it’s just interesting as a comparison.

      • JuanManuel Gimeno Illa

        For me the pattern of managing “unexistence” by means of Optional implies that null can not be accepted as a valid value for “normal” functions. Theh, functions never accept/return null as a valid value. If a function has to signal a nonexistent result its type is T -> Optional.

        And yes, in this case blaming is retroactive: the concept of monad is incompatible with the fact that you allow a null value in your type. If you allow null as a value of the type, null cannot mean “absence of value ” as well.

        In Scala it is not the same: Option(null) that Some(null). The former is the same as java’s Optional.fromNullable and is the correct way to use Option in a context where a null value means absence of value. The second one, allows for a null to be contained in an Option, so null is treated as a value. But you cannot (or shouldn’t) have it both ways !!!

        • Hi again,
          thanks again for your comments.

          I see your point, understand your opinion, though, as you can imagine, I share it only in part.

          Starting from the end, as you pointed out in Scala you have two ways to build an Option, so you can choose if you’d like to allow null as a value or not. The difference here is that you do have that possibility.
          If you meant this double “factory method” (a different behavior for the apply methods of the base and inherited classes) is confusing: it can be. Though I rarely saw the former (i.e. `Option(x)`) used at all.

          I might be missing something in your reasoning about retroactive blaming.
          I think you are right if you mean that null references is the ultimate cause of those errors, just not “the culprit”.

          If you mean that the null reference is bad, of course, no arguing with that (the billion dollar mistake…).
          On the other hand, since I know it’s there, my take is that it’s up to me making sure whatever I design work with it, or ignore it. (At least until we get rid of null – which will never happen).
          Since Java has worked like this for 30 years, it’s up to us to adapt our design to cope with it. IMHO, of course.
          Let me try to explain what I mean with an extreme comparison. You might argue that gravity is sometimes bad or at least inconvenient. But if I throw my TV out of the window, gravity is what causes it to smash into my neighbors’ car. However it’s not gravity I can blame for not having a TV anymore. (Nor, likely, for being beaten up by my neighbor…)

          Right back to null and Optional. Absolutely, you shouldn’t mix null and Optional. That’s a bad practice and, after all, Optional was added to replace null-as-empty.
          There might still be problems though: especially with legacy code, you might still get null.
          With containers. Like Map::get, for instance.
          When the value you wrap into Optional comes from legacy or external code, sometimes null might have been used for different purposes than to express the absence of a value. Maybe it was used for a shared var to mean “uninitialized”. Maybe it was used to mark a value or reading as invalid.
          These might be antipatterns, but unfortunately still there is a chance you can find them in code you haven’t written but you have to interact with.
          Ignoring them and assuming it never happens… risky.
          Failing spectacularly with `Optional.of` is of course to be preferred when you assume null will never show up, but even if it’s better than having a silent failure, it still can mean some downtime.

          Making sure you actually can’t get null is of course the ideal solution.
          Anyway, does not allowing null to be directly contained in an Optional completely avoid NullPointerExceptions?
          Unfortunately it does not, as null references might still be contained in subfields of whatever it’s in there.

          List list = new ArrayList();
          list.add(null);
          Optional<List> lOpt = Optional.of(list);
          lOpt.map(val -> val.get(0).toString()); //throws NPE

          Or to use a more concrete example (using the classes in the article)

          Optional aOpt = Optional.of(new Account(1L, null));
          aOpt.map(a -> a.getBalance().getCurrency()); //throws NPE

          We could minimize the risk by changing these methods to return an Optional themselves… but we can never completely avoid it.

          So again, design of a data structure is a matter of tradeoffs.
          As users, we can only make sure we use every DS in the least error-prone way.

          • JuanManuel Gimeno Illa

            Programmers are always able to use any construction in the language in a wrong way. And Optionals, as a NullPointerException avoidance mechanism, are no exception to this rule.

            By the way, I’m not sure if Scala’s two possibilities is good or it can be a source of errors (because allowing an Option to hold a “no value” without being a None feels counterintuitive, and dangerous, to me)

            So the question is, how can we avoid misusing them. We have to understand that:

            * There is the old way of using null to mean “no value”

            * There is the new way of using Option to me “no value”

            and the important thing is that these two ways cannot be mixed up.

            By the way, other uses of null to mean other things (such as “uninitialized”) can be treated similarly. Code using null to mean unitialized will use Optional.empty() to mean this. (IMO this should be avoided as an antipattern, but you can do it)

            So an old function from Integer -> Integer can accept/return nulls.

            On the other hand, new functions which allow for “no values”, must be of type Optional -> Optional.

            Moreover, a new function from Integer -> Integer shall never accept nor return a null value. Only “true” integers are allowed.

            The ideal is to express the restrictions about accepting/returning null values not in the code but in the type system. Optional and T do not mix up.

            Once we have these architectural rules, the only problem we have is new code (using new functions) calling old functions and vice-versa.

            For instance, with these rules, old code which represents a list of integers (some of which can be “nonexistent”) would be:

            List oli = new ArrayList();

            oli.add(1);

            oli.add(null);

            But new code expressing the same idea would be:

            List<Optional> nli = new ArrayList();

            nli.add(Optional.ofNullable(1));

            nli.add(Optional.ofNullable(null));

            So, an old function can act on oli with no problem:

            List oli2 = oli.stream().map(old_fn).collect(toList());

            And a new function can act on nle with no problem as well:

            List<Optional> nli2 = nli.stream().map(new_fn).collect(toList());

            So the problem is, how can we mix? By adding transformation at the seams between the new and the old. For instance

            List oli2 = oli.stream()

            .map(Optional::ofNullable)

            .map(new_fn)

            .map(oi -> oi.orElse(null))

            .collect(toList());

            Summing up. I think we basically agree: Optionals are a new construct in the language and, as all constructs, it can be missused.

            Disclaimer: English is not my native language, and sometimes some of my expressions are stronger than what I mean :-D

          • Thanks once again for the great feedback!
            I totally agree on the idea of setting ground rules and that, as developers, we should make an effort to move away from using null.

        • For me the pattern of managing “nonexistence” by means of Optional implies that null can not be accepted as a valid value for “normal” functions.

          If I understand things correctly this choice is not up to you (or us). A monad embeds one type system in another and if the types in the underlying type system allow null, then the monad’s unit function must be able to deal with it, which ofNullable does.

          • JuanManuel Gimeno Illa

            Optional.ofNullable is not the unit of Optional. Unit never returns an empty Optional which ofNullable can.
            The unit of Optional is Optional.of, which never returns empty. It can throw, because it makes the assumpltion that null is not a proper value of any type.

          • Well, for sure neither should throw then, in order to be a valid unit function ;-)

            Anyway, even Optional::of breaks the left identity law:

            Function f = new Function<String, Optional>() {
            @Override public Optional apply(String str) { return str == null? Optional.of(“”) : Optional.of(str + str); }
            };

            System.out.println(f.apply(null)); // Optional[“”]
            System.out.println(Optional.of(null).flatMap(f)); // throws NPE

            So we’d be back at square one :D

          • JuanManuel Gimeno Illa

            Throwing an exception is the honest response
            The function you use accepts null as a parameter. IMHO it is an invalid function to be used if one is discussing about monads. Accepting null as a value in a type is what creates havoc with monad laws.

          • Hmm…
            First, thanks again for raising this challenging discussion.

            I’m afraid I agree with Nicolai, that’s not up to us to decide what’s a valid function and what’s an allowed value.
            As long as null is a valid value in Java, and functions handling null are legit in Java, a definition of monad unfortunately can’t, or at least shouldn’t, ignore them.
            The monad laws are to be abided by for all functions and all values, so we can’t really say Optional does abide by.

            Of course, I totally agree that, as a good practice, when using Optional we should make sure to avoid both
            – allowing null as a value
            – using functions that handle null in any way, or expect their input to be null

            And that’s indeed the advice I wanted to give (probably not as clearly stated in the article as in the code on github).

            But one thing is restricting our inputs and following those good practices (which can allow us to use Optional and prevent refactoring bugs, or making them harder anyway). Another thing is the fact that, being those values and functions still possible, our code might be screwed up by maintainers and even external code. And yet another thing is the theoretical discussion.

            About Optional::of, we were not discussing what the honest response was, but if Optional::of was a better unit function and if using it instead of ofNullable would make Optional a valid monad (the answers to which, sadly, seems to be a double no).

          • JuanManuel Gimeno Illa

            I think that we are mixing two IMHO incompatible interpretations/uses of Optional:

            One is the idea of an Optional as a container of values of type T. In this interpretation, an Optional can contain a null because null is a valid value for an element of type T.

            The other is the use of Optional to manage the case of a non-existent value of type T. In this interpretation (which is, BTW what occurs in languages such as Haskell), no null value can be inside an Optional. Simply it makes no sense to assert the existence of a value that is non-existant.

            I think, if we want to use Optionals to manage non-existence, we must consider the second interpretation: that null is not a valid value of type T. If we allow it then we cannot use Optionals to manage non-existence of values.

            And, if we want Optionals to have monad semantics, we must restrain ourseleves to functions that treat null as an invalid value of any type.

          • Your reasoning is a valid opinion about how to design the implementation of a Maybe monad. Even a good opinion about how the language should work.
            Unfortunately, we can’t draw any other conclusion from that opinion, as the way Java works (which is a fact) tells us differently.

            You can’t decide what’s a valid value, as long as the language allows it.

            You can have best practices suggesting not mixing null with Optional. But you can’t force people to follow them.

            You can have a design that does not allow null at all. But a few comments above we have seen that, as long as null is part of Java, no Optional implementation ignoring it can abide by monad laws. So that makes “not accepting null” a valid design (better or worse, not for me to say) but does not make it a design satisfying monad semantics.

            As we also saw, you can’t even say that null is always used to express the absence of a value. Antipatters might always be involved, but still, using it with a different meaning is a possibility. Another example might be for a Tree-like data structure, using null to represent leaves, and having functions walking the tree handle null as a special case. Bad practice, terrible design: no doubt! But still possible and legal.

            You mention Haskell… but Haskell doesn’t have null references at all. So it’s hard to draw a comparison with Java. Yes, maybe that’s the way it’s supposed to work – but Java is not Haskell, for good or for bad.

            I think you clearly explained your point of view by now.
            Maybe you could provide a few links to useful readings to examine the topic in further depth?
            Something about monad unit function being forbidden to return empty, but being allowed to throw?
            Thanks

          • JuanManuel Gimeno Illa

            Trying to condense my argument: if one wants to use Optional to manage null, one has to forbid functions to use null as a value. So one has to go, if you want, against one of the things that Java allows. If you want to continue using null as a value of any type you can, but then don’t try to use Optional. But then, whether Optional breaks monad laws or not, IMHO, it’s not a problem at all.

  • A few notes after gathering some (very productive) feedback.

    1) This post is not meant to blame Optional or its design.
    It recognizes that design choices are always a trade-off, and not allowing null for non-empty Optionals has PROs and CONs.
    2) It shows some CONs (namely breaking monad laws and preventing refactoring “without fear”).
    3) It DOES NOT suggest to avoid using Optional, at all! Rather…
    4) …The errors shown shown are all revolving around null and functions handling null, very specific situations. So the article offers advice on how to prevent these possibly dangerous situations.
    In particular:
    – Be careful when values wrapped into Optional can be null (and try to make sure this won’t happen)
    – Make sure functions used in a map chain won’t handle null as a special case
    – Prefer flatMap over map, and prefer functions returning Optional
    – Be careful when refactoring by composing functions (see below how apply and flatMap are used together – the way we define the functions should drive anyone refactoring the chain to do so)

    Overall, referring to our example, one of the best pattern we could think of to help safely using Optional was the following:

    “`
    Map bank = new HashMap();
    Function<Long, Optional> findAccount = id -> Optional.ofNullable(bank.get(id));

    Function<Account, Optional> extractBalance = account -> Optional.ofNullable(account.getBalance());

    Function<Balance, Optional> toDollars = balance -> {
    switch (balance.getCurrency()){
    case DOLLAR:
    return Optional.ofNullable(balance.getAmount());
    case POUND:
    return Optional.ofNullable(balance.getAmount() * 1.3);
    case EURO:
    return Optional.ofNullable(balance.getAmount() * 1.1);
    default:
    return Optional.ofNullable(0.);
    }
    };

    Optional accountId4 = Optional.of(4L);

    bank.put(4L, null);

    System.out.println(accountId4.flatMap(findAccount).flatMap(extractBalance).flatMap(toDollars)); // Optional.empty
    System.out.println(accountId4.flatMap(id ->
    findAccount.apply(id).flatMap(account ->
    extractBalance.apply(account)).flatMap(toDollars))); // Optional.empty

    “`

    Please check on github for a more thorough discussion focused on code examples, and further explanations.
    https://github.com/mlarocca/sitepoint/blob/master/Java/Optional/Test.java#L386

  • volothamp

    Marcello can you check the examples? The first one gives a different result

    “`
    Function<Integer, Optional> f = x -> Optional.ofNullable(
    x == null
    ? -1
    : (x == 2 ? null : x + 1));

    // true, Optional[1] === Optional[1]
    Optional left = Optional.of(1).flatMap(f);
    Optional right = f.apply(1);

    // This prints Optional[2]
    System.out.println(left);
    System.out.println(right);
    System.out.println((left.equals(right)));
    “`

    • Good catch! :)
      Thanks for sharing.

      Indeed, when I changed that function `f` for the example I forgot to update the comment. It was even worse for the second example, as the nested ternary operator causes a NPE trying to autobox null.

      I’ll have it (and another nit) fixed asap, thanks again!

  • Hi Alejandro,
    thanks for the link, the article seems indeed pretty interesting. At a first glance, most of it makes sense to me.
    Let me read it more carefully and I’ll get back to you later today with a (hopefully) more insightful answer.

    Cheers!

    • So, indeed, there is sound advice in the Colebourne’s post.
      The most important one is: make sure you always return an Optional in your methods when a value is not always present – never use null.
      Enforcing a good, clean and easy-to-use interface should be our first concern, as it’s easier to make mistakes when integrating external code.
      As he says, reducing the number of places where null could be returned will, in time,

      It is, indeed, not advised to “always” replace null with Optional, and class attributes are a good example of when the advantage is limited. Using null internally can be fine, as you are going to be the only one meddling with it (and for teams and future maintenance, you can make up with clear comments).
      You’d have several advantages, starting from making you class Serializable (one hiccup we haven’t mentioned in this article) to avoiding the overhead associated with Optional.

      Moreover, using optional for return types in every function used in a Optional-map chain, will allow you to avoid refactoring hazards, as discussed in this article and repo (https://github.com/mlarocca/sitepoint/blob/master/Java/Optional/Test.java#L386)

      About parameters for methods and constructors, that’s a good point: if callers have a the raw value they can use the method directly, otherwise, if they have an Optional, they can always use flatMap on it.
      As an alternative, polymorphism comes to the rescue, and a convenient version taking Optionals can be provided in addition to the basic one.

      But the best piece of advice, in my opinion, is this one: “The key to making this approach work beyond the basics is to learn the various methods on Optional. If you simply call Optional.get() you’ve missed the whole point of the class.”

      And indeed several other developers suggest not using Optional just to wrap&get values., working like a conditional in some ways.
      BTW, let me forward you this presentation by Mark Stuart that our awesome editor, Nicolai (who, btw, was absolutely instrumental in the writing of this article) had suggested to me some time ago: https://www.youtube.com/watch?v=fDvH15PffoE&t=4h4m

      Finally, from a FP point of view… that pattern doesn’t seem expressly FP-oriented, but neither in contrast with it.
      In summary, if you combine those good practices with the ones we mentioned, you should be able to have clean and safe code, so by all means!

      Even better, if you can use external libraries, apply those patterns in the Colebourne’s’s article and JavaSlang’s Option instead of java.util.Optional (make sure to use Option.some to create an Option)

      • Alejandro Gervasio

        Hi Marcello,

        Well, first of all, thanks for taking the time to read the Colebourne’s post and share your opinions here. By no means I want to sound way too flowered in my comments, but anyone wanting to start out using Optional, based on a solid programming style should read your post and these guidelines. I totally agree with you in every point you raised up above. BTW, I’ll look at Mark Stuart’s presentation.

        Have a good time!

        • Thanks Alejandro, it’s really kind of you :)

          Stay tuned, we might have a follow up soon, and there are many other super-interesting articles on the ramp for this channel as well.

          So long

          • Alejandro Gervasio

            Thanks Marcello for the update. Looking forward to see them :)

            So long

Recommended
Sponsors
Get the latest in Java, once a week, for free.