[TODO: Finish out voteForRemoval() with EasyMock. Run some comparisons with JMock/NMock and RhinoMocks]
These days, I don't like mock (or stub, fake, dummy, shunt, whatever) frameworks much. Frankly, I've only worked with one mock library, and it wasn't one of the cool kids on the block because I needed something that was JDK 1.1 compatible***. We ported the app in question to C#, and the mock library couldn't come along for the ride. By that time, I'd already started falling out of love with the framework for two reasons:
1) The code in my test to setup the mock seemed unnatural, and too easy to forget to tell the mock to kick in and do its thing. It felt like it interrupted the flow of things and didn't leave the most readable test.
2) When the mock threw an exception, it was never really obvious what went wrong. By the time I'd traced down my problem, many times it was a flaw in configuration of the mock, not a helpful hint to a real problem. I suppose error reporting could be better in other frameworks than the one I'd worked with, but I'm not sure given the inherent way the framework must work.
(I attended a recent presentation on NUnit and RhinoMocks by a DfwPPer I met recently, David O'Hara, and as far as I'm concerned, his wrangling with RhinoMocks simply confirmed my beliefs).
Any test that I needed to get caught up to snuff during the port, I'd simply replace the mocks with real objects in many cases, or roll my own mock, and I don't recall having suffered much with this approach.
So ... I'm going to put myself to the test. EasyMock is recommended in the Pragmatic Unit Testing book, and EasyMock comes with some sample code. I'm using EasyMock 2.3, JUnit 4 in Eclipse 3.3 configured for Java 6, loaded up with the code inside samples.zip in the EasyMock distribution. (Keep in mind, EasyMock is an innocent bystander in this article as I'm addressing mock frameworks in general).
EasyMock code reproduced here under its license**
First, I'll make a copy of ExampleTest called ExampleTestDiyMock, and remove all of the imports for EasyMock:
package org.easymock.samples;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;
import org.junit.Before;
import org.junit.Test;
public class ExampleTestDiyMock {
...
Setup is broken now, since it can't create the mock. I'll replace it with my own mock class. Setup now looks like:
@Before
public void setup() {
mock = new CollaboratorMock();
classUnderTest = new ClassUnderTest();
classUnderTest.addListener(mock);
}
and Eclipse creates this default CollaboratorMock class for me:
package org.easymock.samples;
public class CollaboratorMock implements Collaborator {
@Override
public void documentAdded(String title) {
}
@Override
public void documentChanged(String title) {
}
@Override
public void documentRemoved(String title) {
}
@Override
public byte voteForRemoval(String title) {
return 0;
}
@Override
public byte voteForRemovals(String[] title) {
return 0;
}
}
Next, the removeNonExistingDocument() test isn't compiling. It's a short test and I'm trying to figure out exactly what gets asserted:
@Test
public void removeNonExistingDocument() {
replay(mock); // <- The method replay(Collaborator) is undefined for the type ExampleTestDiyMock
classUnderTest.removeDocument("Does not exist");
}
A review of the EasyMock readme has some comments in the code...:
public void testRemoveNonExistingDocument() {
// 2 (we do not expect anything)
replay(mock); // 3
classUnderTest.removeDocument("Does not exist");
}
... and this explanation, “After activation in step 3, mock is a Mock Object for the Collaborator interface that expects no calls. This means that if we change our ClassUnderTest to call any of the interface's methods, the Mock Object will throw an AssertionError”
Right away, this is a small example of, to my mind, an unintuitive bit of code for the test. If you want to assert that nothing is called on our mock, don't tell it to do anything and call replay. It makes sense within the context of how the framework works, but the context of the framework appears backwards to how I want to think of the test.
Digging into the class under test's code for removeDocument, if it can't find the string in its list of documents, it should not call the Collaborator's documentRemoved method.
Here's how I code up the test for my home-rolled mock:
@Test
public void removeNonExistingDocument() {
mock.throwIfDocumentRemovedIsCalled();
classUnderTest.removeDocument("Does not exist");
}
along with the implementation inside my homemade mock:
public class CollaboratorMock implements Collaborator {
private boolean throwIfDocumentRemovedIsCalled;
...
@Override
public void documentRemoved(String title) {
if (throwIfDocumentRemovedIsCalled) {
throw new RuntimeException("documentRemoved was called when it shouldn't have been.");
}
}
...
public void throwIfDocumentRemovedIsCalled() {
this.throwIfDocumentRemovedIsCalled = true;
}
}
Test passes. I'll break the production code on purpose and see what the failures look like with my mock and EasyMock's. I change this code:
public class ClassUnderTest {
...
public boolean removeDocument(String title) {
if (!documents.containsKey(title)) {
return true;
}
...
}
...
}
by simply removing the not operator:
public boolean removeDocument(String title) {
if (documents.containsKey(title)) {
I'm not sure if that will really do it, but looks like it may, so I give it a shot. Interestingly, my test still passes, which is unexpected. I look at the production code and see the next bit after the check for documents.containsKey is a call to listenersAllowRemoval. I guess that my default behavior in my mock is to not allow removal, which is why the test still passes.
I run the original EasyMock test, and it fails. Here's its failure stack trace:
java.lang.AssertionError:
Unexpected method call voteForRemoval("Does not exist"):
at org.easymock.internal.MockInvocationHandler.invoke(MockInvocationHandler.java:29)
at org.easymock.internal.ObjectMethodsFilter.invoke(ObjectMethodsFilter.java:56)
at $Proxy5.voteForRemoval(Unknown Source)
at org.easymock.samples.ClassUnderTest.listenersAllowRemoval(ClassUnderTest.java:80)
at org.easymock.samples.ClassUnderTest.removeDocument(ClassUnderTest.java:37)
at org.easymock.samples.ExampleTest.removeNonExistingDocument(ExampleTest.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
My first reaction is “what's up with voteForRemoval? It says it's not an expected call - is that a bad thing?”. Then, remembering the production code first checks with the Collaborators to see if removal is alright, this failure makes sense.
Things are in a funky state now, not really testing the thing we want to test, and arguably EasyMock has an advantage because with this wonky state of things, it's at least breaking, whereas my test thinks everything is fine -- though perhaps in a literal sense, my test is fine because documentRemoved isn't being called. [Bookmark A].
I'll improve my test by also checking for no calls to voteForRemoval.
@Test
public void removeNonExistingDocument() {
mock.throwIfDocumentRemovedIsCalled();
mock.throwIfVoteForRemovalIsCalled();
classUnderTest.removeDocument("Does not exist");
}
And now it fails:
java.lang.RuntimeException: voteForRemoval was called when it shouldn't have been.
at org.easymock.samples.CollaboratorMock.voteForRemoval(CollaboratorMock.java:26)
at org.easymock.samples.ClassUnderTest.listenersAllowRemoval(ClassUnderTest.java:80)
at org.easymock.samples.ClassUnderTest.removeDocument(ClassUnderTest.java:37)
at org.easymock.samples.ExampleTestDiyMock.removeNonExistingDocument(ExampleTestDiyMock.java:26)
EasyMock is proving itself to be cleaner so far and less work (assuming you're familiar with the framework, which I will), but personally I think my test is more readable.
Both tests bug me though because I'm used to seeing assert somewhere in a unit test. I'll change my approach:
public class CollaboratorMock implements Collaborator {
private boolean documentRemovedCalled;
private boolean voteForRemovalCalled;
...
@Override
public void documentRemoved(String title) {
this.documentRemovedCalled = true;
}
@Override
public byte voteForRemoval(String title) {
this.voteForRemovalCalled = true;
return 0;
}
...
public boolean wasDocumentRemovedCalled() {
return documentRemovedCalled;
}
public boolean wasVoteForRemovalCalled() {
return voteForRemovalCalled;
}
}
The test now reads:
@Test
public void removeNonExistingDocument() {
classUnderTest.removeDocument("Does not exist");
assertFalse(mock.wasDocumentRemovedCalled());
assertFalse(mock.wasVoteForRemovalCalled());
}
And the failure is a typical assertion stack trace, which in this case, isn't as readable as either of the prior stack traces (mine or EasyMock's). I could improve the readability of the assert in the stack trace by using the message parameter. The test itself is a lot better for my eyes. It's clear what I'm doing and what I'm asserting. I'm not familiar with EasyMock, but perhaps after sometime, I could learn to appreciate what no setup and a call to replay(mock) means.
Another thought at this point on the value of the test if the production code undergoes enhancements in the future. The EasyMock test will fail if any new Collaborator method is added and called, whereas my test would probably be oblivious to the new functionality and continue to roll on -- which is actually the exact state of things back up where I put [Bookmark A] in this article. We could pretend that both tests were passing, then someone added the voteForRemoval functionality into the Collaborator interface. Which is more valuable, the test that's tightly focused on just the calls to Collaborator we're trying to verify (mine) or the test that's verifying that NO calls are made to the Collaborator at all ever in the case of trying to remove a document that doesn't exist (EasyMock)?
In this specific case, I'd say EasyMock is the better test here, simply because I believe I wouldn't want my Collaborators bothered with a voteForRemoval for a document we're not even going to remove. I'd be glad the EasyMock test blew up and gave me the chance to fix that problem. And trying to do the same with my homemade mock would probably result in a mess of code.
But what if the new functionality was a new Collaborator method called documentForRemovalNotFound? Then I'd probably change my mind and say, dang it, EasyMock, it's actually not a problem -- your job on this test is to guard against the documentRemoved call, not any other calls - I'll have other tests worry with that.
Well ... tangent over, I'll press on.
[For what it's worth, it's not necessary, but a verify(mock) call can be added to the end of the EasyMock removeNonExistingDocument test, and that helps my brain a bit with readability].
First off, I restore ClassUnderTest and make sure all the tests are passing.
I'm getting pressed for time, so I'll skip the addDocument() test, looks like I may end up with a similar looking test anyway. The addAndChangeDocument() test looks interesting:
@Test
public void addAndChangeDocument() {
mock.documentAdded("Document");
mock.documentChanged("Document");
expectLastCall().times(3);
replay(mock);
classUnderTest.addDocument("Document", new byte[0]);
classUnderTest.addDocument("Document", new byte[0]);
classUnderTest.addDocument("Document", new byte[0]);
classUnderTest.addDocument("Document", new byte[0]);
verify(mock);
}
As I study this, I'm starting to think this sample code may not be a good test bed to play with here for my article. If I were writing this test, I wouldn't care about testing the repetition, two calls to addDocument would suffice. I'd also want a method in the production class to retrieve a document so I could verify the contents of it. But obviously, the job of this sample code is to help demonstrate different features of the mock library, and the sample code may suffer a bit for my purposes.
I look on to the voteForRemovals test ... I think my mock may suffer more in addAndChangeDocument test, but may do better in voteForRemovals test. I'll try to play fair and do both. For the sake of sleep, I'm going to be less verbose in my code changes here. [code ... code ... code]. Okay. Here's what my mock class looks like now:
package org.easymock.samples;
public class CollaboratorMock implements Collaborator {
private boolean documentRemovedCalled;
private boolean voteForRemovalCalled;
private String lastDocumentAdded;
private String lastDocumentChanged;
@Override
public void documentAdded(String title) {
lastDocumentAdded = title;
}
@Override
public void documentChanged(String title) {
lastDocumentChanged = title;
}
@Override
public void documentRemoved(String title) {
this.documentRemovedCalled = true;
}
@Override
public byte voteForRemoval(String title) {
this.voteForRemovalCalled = true;
return 0;
}
@Override
public byte voteForRemovals(String[] title) {
return 0;
}
public boolean wasDocumentRemovedCalled() {
return documentRemovedCalled;
}
public boolean wasVoteForRemovalCalled() {
return voteForRemovalCalled;
}
public String getLastDocumentAdded() {
return lastDocumentAdded;
}
public String getLastDocumentChanged() {
return lastDocumentChanged;
}
public void reset() {
documentRemovedCalled = false;
voteForRemovalCalled = false;
lastDocumentAdded = null;
lastDocumentChanged = null;
}
}
And my new homemade version of the addAndChangeDocument test:
@Test
public void addAndChangeDocument() {
assertEquals(null, mock.getLastDocumentAdded());
classUnderTest.addDocument("Document", new byte[] { 0x01 } );
assertEquals("Document", mock.getLastDocumentAdded());
for (int i = 0; i < 3; i++) {
mock.reset();
assertEquals(null, mock.getLastDocumentChanged());
assertEquals(null, mock.getLastDocumentAdded());
classUnderTest.addDocument("Document", new byte[] { 0x02 });
assertEquals(null, mock.getLastDocumentAdded());
assertEquals("Document", mock.getLastDocumentChanged());
}
}
A quick look at both tests, EasyMock looks cleaner, especially without the looping that I put in my test. My mock code has grown a bit, and calling that chunk 3 times in a row only reads well with the for loop - the EasyMock simply calls the same line over and over and it's nice and clean without a for loop.
What about readability? Perhaps this is my own hangup, but the lines in the EasyMock test that call documentAdded() and documentChanged() on the mock instance throw me. My dense brain may have a conversation with itself like the following: “Lessee, I'm notifying the mock that a document has been added ... okay, why am I doing that? ... oh, wait, this is how I tell the mock what to expect - right.” Another example of the mental shift in context that a mock framework requires of me.
My code is bulkier, but reads more straightforward in my opinion. “Okay, assert the last document title added is null, makes sense, we're starting the test, nothing's happened yet. Next, add the document, then assert the last title added matches. Cool. Okay ... for loop ... mock.reset(). Hmm? What does that mean?” ... here I've potentially done no better than a mock framework. Then the rest would read as the first part did.
I'll break the production code and see how the tests do. Again, I'll just flip the boolean result inside addDocument such that it will call documentChanged when it's added and documentAdded when it's changed.
public void addDocument(String title, byte[] document) {
boolean documentChange = !documents.containsKey(title); // <-- bug introduced here
documents.put(title, document);
if (documentChange) {
notifyListenersDocumentChanged(title);
} else {
notifyListenersDocumentAdded(title);
}
}
The EasyMock test gives this stack:
java.lang.AssertionError:
Unexpected method call documentAdded("Document"):
documentAdded("Document"): expected: 1, actual: 1 (+1)
documentChanged("Document"): expected: 3, actual: 1
at org.easymock.internal.MockInvocationHandler.invoke(MockInvocationHandler.java:29)
at org.easymock.internal.ObjectMethodsFilter.invoke(ObjectMethodsFilter.java:56)
at $Proxy5.documentAdded(Unknown Source)
at org.easymock.samples.ClassUnderTest.notifyListenersDocumentAdded(ClassUnderTest.java:68)
at org.easymock.samples.ClassUnderTest.addDocument(ClassUnderTest.java:28)
at org.easymock.samples.ExampleTest.addAndChangeDocument(ExampleTest.java:54)
and we can click into line 54 in the test:
@Test
public void addAndChangeDocument() {
mock.documentAdded("Document");
mock.documentChanged("Document");
expectLastCall().times(3);
replay(mock);
classUnderTest.addDocument("Document", new byte[0]);
classUnderTest.addDocument("Document", new byte[0]);
classUnderTest.addDocument("Document", new byte[0]); // <-- here ... really
classUnderTest.addDocument("Document", new byte[0]);
verify(mock);
}
My test gives this one:
java.lang.AssertionError: expected:<Document> but was:<null>
at org.junit.Assert.fail(Assert.java:71)
at org.junit.Assert.failNotEquals(Assert.java:451)
at org.junit.Assert.assertEquals(Assert.java:99)
at org.junit.Assert.assertEquals(Assert.java:116)
at org.easymock.samples.ExampleTestDiyMock.addAndChangeDocument(ExampleTestDiyMock.java:42)
jumping to line 42 of the test:
@Test
public void addAndChangeDocument() {
assertEquals(null, mock.getLastDocumentAdded());
classUnderTest.addDocument("Document", new byte[] { 0x01 } );
assertEquals("Document", mock.getLastDocumentAdded()); // <-- here
assertByteArrayEquals(new byte[] { 0x01 }, classUnderTest.getDocument("Document"));
...
I'm staring at the EasyMock output. First off, the error message is interesting:
Unexpected method call documentAdded("Document"):
documentAdded("Document"): expected: 1, actual: 1 (+1)
documentChanged("Document"): expected: 3, actual: 1
documentAdded ... expected: 1, actual: 1 (+1) -- what does that mean? [Especially if I'm just reading this after a build machine run and not while I've my head in the code.] “If expected and actual are both 1, why is it failing? Maybe the +1 means it was called twice ... ? Then it expected 3 calls to documentChanged, but only got one. So ... the bug must be that we should've called documentChanged 2 more times? Wait - “Unexpected method call documentAdded” - but it was expecting one. And it got that one. Okay, it must have gotten a second call it wasn't expecting. Well, lemme go to the stack and find where in the test code it failed ... line ... 54. 54? Okay, so on the third call to addDocument in my test code it blew up. Lemme look at the production code. Weird, the boolean is flipped incorrectly, so on the first call to addDocument, that code was calling documentChanged when it should have called documentAdded. That's even more confusing, the stack trace (line 54? not line 52? ... yeah line 54 - huh) says the test fails on the third call to addDocument. That's strange.”
Leaving my mock conversation in my head (har) ... this is still genuinely strange. I wasn't expecting this sort of example to come up, and perhaps once I've vetted this after some sleep I'll find I'm doing something wrong here, but this feels like most any other mock framework stack trace I've had to sort through (and the ones David O'Hara had to sort through on the fly when his presentation almost got away from him :-) It was a nice save, though :-). What I may gain in writing test code (which is debatable to me), I can lose in maintaining the dang thing.
With my failure trace and where the stack trace jumps me to my test code and my homemade mock, it's obvious. I called addDocument, was expecting the document title to show in my mock collaborator, and the mock says, “Nope - it's still null.” Problem was in the first call to addDocument - I dig in, find the incorrect boolean evaluation, fix it, re-run the test and I'm gone.
Well, I'm off for now. I still want to come back and try voteForRemoval().
If you, gentle reader, have something to add to the conversation, esp. if I've mussed up proper usage of EasyMock or have lopsided examples or think I should be working with a different mock library - shoot me a comment or an email.
** EasyMock license
EasyMock 2 License (MIT License)
Copyright (c) 2001-2007 OFFIS, Tammo Freese.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*** This line received a critique that ran something like “You get an F for logic or an A+ for balls.” Genius :-) I replied that it was probably both. In my own not-so-humble interpretation of what I've written here, I've tried to make it clear that while my favor for DIY mocks is a strongly held opinion of mine, I realize it may not stand up in light of frameworks I've not worked with -- which is half the reason I wrote this, to get feedback from others.
tags: ComputersAndTechnology
