In the first part of this series, I learned how to use Macro Annotations to add a few simple methods to a class and fill
in the implementation of a trait.
My eventual goal is to be able to take a trait which defines a few basic Int
and Float
fields,
and automatically generate a class that would read those fields from a ByteBuffer
. Eg., given this:
trait A { | |
def x: Int | |
def y: Float | |
} |
My macro should generate the equivalent of:
class B(bb: ByteBuffer) extends A { | |
def x = bb.getInt(0) | |
def y = bb.getFloat(4) | |
} |
In the previous post, I was only able to add the methods for one fixed trait – not very useful. If I was going to be able to fill in the implementation for any trait, I’d need to use scala reflection to figure out which methods I needed to add. In this post, we’ll learn scala reflection and then generalize our macro to work on any trait.
Once again, I’d like to warn the reader that is not so much a tutorial, as a journal of my path to discovery, with some mistakes and detours along the way.
Reflection Basics
I started out by learning the basics of scala reflection. It’s pretty easy to play around with reflection in the repl. Let’s start with our same set of imports from last time:
import language.experimental.macros | |
import reflect.macros.Context | |
import scala.annotation.StaticAnnotation | |
import scala.reflect.runtime.{universe => ru} |
The first step when using reflection is getting a handle on the Type
of something, using the typeOf
method from our universe. Once we have a type, we can ask for lots
of information, like all methods and variables. (If you’re familiar with reflection in Java, this should all be similar in spirit.)
scala> trait A { | |
| def x: Int | |
| def y = 7.0f | |
| } | |
defined trait A | |
scala> val typ = ru.typeOf[A] | |
typ: reflect.runtime.universe.Type = A | |
scala> typ.members | |
res0: reflect.runtime.universe.MemberScope = | |
Scopes(method y, method $init$, method $asInstanceOf, method $isInstanceOf, ... | |
scala> typ.members.filter{_.isMethod}.map{_.asMethod} | |
res1: Iterable[reflect.runtime.universe.MethodSymbol] = | |
List(method y, method $init$, method $asInstanceOf, method $isInstanceOf, ... |
Note that the MemberScope
returned by typ.members
is a Traversable
, so I can call map
, filter
, etc. Here I’ve filtered down to just the methods, and
also “cast” them to methods with asMethod
.
I only needed to do some small filtering on top of this. I wanted to filter down to only those methods that (1) take no arguments, (2) return either an Int
or a
Float
, and (3) that are undefined. I was able to write some fairly straight-forward utility methods to handle the first two:
def zeroArgMethods(typ: ru.Type) = { | |
typ.members.collect{ | |
case m if m.isMethod => m.asMethod | |
}.filter{_.paramss.isEmpty} | |
} | |
def targetMethods(typ: ru.Type) = { | |
zeroArgMethods(typ).filter{m => | |
val rt = m.returnType | |
(rt =:= ru.typeOf[Int]) || (rt =:= ru.typeOf[Float]) | |
} | |
} |
I could use these on the type of my example trait, and filter out most of the methods I should ignore:
scala> zeroArgMethods(typ) | |
res8: Iterable[reflect.runtime.universe.MethodSymbol] = | |
List(method y, method asInstanceOf, method isInstanceOf, method x) | |
scala> targetMethods(typ) | |
res9: Iterable[reflect.runtime.universe.MethodSymbol] = | |
List(method y, method x) |
I only had one problem left. I wanted my macro to only supply an implementation for undefined methods. My example trait supplied an implementation for y
, but I hadn’t filtered it out yet.
Detour: Using Reflection on Methods
There is actually a very simple solution to finding defined and undefined methods. But first, I got the crazy idea of using reflection to discover how to use reflection. This is section is totally unnecessary for my end goal, but hopefully you’ll find it interesting nonetheless.
I needed to find some way to differentiate method x
from method y
. My first instinct was to just look at what was available to me with tab-completion,
hoping something would look like isDefined
or isAbstract
.
scala> val mX = targetMethods(typ).find{_.name.toString == "x"}.head | |
mX: reflect.runtime.universe.MethodSymbol = method x | |
scala> val mY = targetMethods(typ).find{_.name.toString == "y"}.head | |
mY: reflect.runtime.universe.MethodSymbol = method y | |
scala> mX. | |
NameType accessed allOverriddenSymbols alternatives annotations asClass | |
asFreeTerm asFreeType asInstanceOf asMethod asModule asTerm | |
asType associatedFile companionSymbol filter fullName getter | |
isAbstractOverride isAccessor isByNameParam isCaseAccessor isClass isConstructor | |
isErroneous isFinal isFreeTerm isFreeType isGetter isImplementationArtifact | |
isImplicit isInstanceOf isJava isLazy isLocal isMacro | |
isMethod isModule isModuleClass isOverloaded isOverride isPackage | |
isPackageClass isParamAccessor isParamWithDefault isParameter isPrimaryConstructor isPrivate | |
isProtected isPublic isSetter isSpecialized isStable isStatic | |
isSynthetic isTerm isType isVal isVar isVarargs | |
map name newClassSymbol newMethodSymbol newModuleAndClassSymbol newTermSymbol | |
newTypeSymbol orElse owner paramss privateWithin returnType | |
setter suchThat toString typeParams typeSignature typeSignatureIn |
Hmm, scanning the list, i didn’t see anything that jumped out at me. The list of methods was too long to go through them all by hand.
But then I realized – I could use reflection to call of those methods on both x
and y
, and see which ones yielded different results. That is,
I’d be getting MethodSymbol
s that were members of MethodSymbol
. (Yes, there was probably an easier way, but now this just sounded fun.)
To call methods using reflection, you first need to get a runtime mirror, from that get an instance mirror, and then finally from there you can call the methods:
/** | |
* this methods finds all the zero-arg methods of the argument, and calls | |
* them using reflection. It returns a map of the results | |
*/ | |
def callAllZeroArgMethods[A:reflect.runtime.universe.TypeTag: reflect.ClassTag](a: A): | |
Map[reflect.runtime.universe.MethodSymbol, util.Try[Any]] = { | |
val zms = zeroArgMethodsOf[A](a) | |
//first get the runtime mirror ... | |
val cm = ru.runtimeMirror(ru.getClass.getClassLoader) | |
//then the instance mirror ... | |
val aMirror = cm.reflect(a) | |
// and now, for each of the methods ... | |
zms.map {zm => | |
//call the method, wrapped in a Try | |
zm -> util.Try{aMirror.reflectMethod(zm)()} | |
}.toMap | |
} | |
import ru._ //not really sure why this is necessary ... but otherwise can't find ClassTags for ru.MethodSymbol | |
/** | |
* this method calls all zero-arg methods of two different methods, and compares | |
* the results. Mostly its normal scala-code to nicely format the results | |
*/ | |
def diffZeroArgMethods(methodA: MethodSymbol, methodB: MethodSymbol) = { | |
def c(m: Map[MethodSymbol,util.Try[Any]]): Map[String,String] = { | |
m.map{case (k,v) => k.toString -> {v match { | |
case util.Success(s) => Option(s).toString | |
case util.Failure(f) => "<xxx>" | |
}}} | |
} | |
val as = c(callAllZeroArgMethods(methodA)) | |
val bs = c(callAllZeroArgMethods(methodB)) | |
val fmt = "%20s|%40s%40s" | |
println(fmt.format("", methodA, methodB)) | |
println("-" * 100) | |
as.filter{case(k,v) => bs(k) != v}.foreach{ case(method,aVal) => | |
println(fmt.format(method, aVal, bs(method))) | |
} | |
} |
Now I could run this on x
and y
to figure out how to find an abstract method:
scala> diffZeroArgMethods(mX,mY) | |
| method x method y | |
---------------------------------------------------------------------------------------------------- | |
method returnType| Some(scala.Int) Some(scala.Float) | |
method fullName| Some($line15.$read.$iw.$iw.$iw.$iw.A.x) Some($line15.$read.$iw.$iw.$iw.$iw.A.y) | |
method asMethod| Some(method x) Some(method y) | |
method asTerm| Some(method x) Some(method y) | |
method name| Some(x) Some(y) | |
method typeSignature| Some(=> scala.Int) Some(=> scala.Float) | |
method alternatives| Some(List(method x)) Some(List(method y)) |
That was a little disappointing. The only differences were from the name or the method type. I was expecting a method that returned true
for x
and false
for y
(or vice versa). What gives?
So I turned to stackoverflow. Turns out it was an oversight in the reflection api, but there is a workaround.
def isDefined(method: u.MethodSymbol): Boolean = { | |
//from http://stackoverflow.com/questions/16792824/test-whether-a-method-is-defined | |
!isDeferred(method) | |
} | |
def isDeferred(sym: u.MethodSymbol) = { | |
sym.asInstanceOf[scala.reflect.internal.Symbols#Symbol]. | |
hasFlag(scala.reflect.internal.Flags.DEFERRED) | |
} |
Nonetheless, that was a fun little experiment in using reflection. (I warned you it was a detour!)
Runtime and Compile Time Universes
So far I’ve been using runtime reflection as an easy way to learn. But to use in my macros, I’d need to switch to compile time reflection. For the most part, reflection is the same, but you use a different Universe
in compile time reflection. Since all of the types are dependent on the universe, this also means all the types of my methods change.
I really wanted to have common methods which worked for both runtime & compile time reflection, and which I could unit test. To support both, I put all of my methods into a helper class which takes a scala.reflect.api.Universe
(the parent to runtime & compile time universes). Then I can instantiate it with either scala.reflect.runtime.universe
or context.universe
in a macro.
The full code is checked into my learn_macros project, even with some simple unit tests.
Parameterizing Annotations
I’ve got the basics of reflection – now I just needed to get a handle on the Type
of my target trait inside my macro annotation. This one was surprisingly difficult to figure out, I had to turn to stackoverflow and got the answer from Eugene Burmako.
When using my macro annotation, I make the target trait a type parameter on the annotation class, and then pull it out of the type with a call to typeCheck
:
//the annotation class has type parameter T | |
class FillDefsWithReflection[T] extends StaticAnnotation { | |
def macroTransform(annottees: Any*) = macro FillDefsWithReflectionImpl.impl | |
} | |
object FillDefsWithReflectionImpl { | |
def impl(c: Context)(annottees: c.Expr[Any]*): c.Expr[Any] = { | |
import c.universe._ | |
//we need to do some pattern matching to pull out just the type of the trait | |
val targetTrait = c.prefix.tree match { | |
case Apply(Select(New(AppliedTypeTree(Ident(_), List(typ))), nme.CONSTRUCTOR), List()) => typ | |
} | |
//of course, 7 is not really an instance of our target trait -- but we | |
// don't care; we just want it to *type check* as our target trait, | |
// so we can pull out the type | |
val tpe = c.typeCheck(q"(7.asInstanceOf[$targetTrait])").tpe | |
... | |
} | |
} |
Then the annotation can be used like so:
@FillDefsWithReflection[MyTrait] class Blah |
Again, I put together some unit tests to verify the behavior.
Putting It All Together
We’ve got all the key pieces now. We know how to:
- Template our macro annotation with the type of trait it should add.
- Use reflection inside our macro to find all the getters it needs to define.
- Add methods to the existing class defintion (from part 1).
Since what’s left is mostly normal scala coding, I won’t go through it in
detail here. But you can take a look at the full implementations. There are two versions of my annotation, @ByteBufferBacked
with just getters, and @MutableByteBufferBacked
which also adds in setters, which can be used like so:
//first we define our trait | |
trait PersonalInfo { | |
def height: Float | |
def weight: Float | |
def phoneNumber: Int | |
def birthYear: Int | |
} | |
//now we can create a concrete class that knows how to read all | |
// the fields in a ByteBuffer | |
@ByteBufferBacked[PersonalInfo] | |
class PersonalInfoImpl(val bb: ByteBuffer) | |
//and we can also make a mutable version, which includes setters | |
@MutableByteBufferBacked[PersonalInfo] | |
class MutablePersonalInfoImpl(val bb: ByteBuffer) | |
//the classes now just behave like normal implementation of the trait: | |
def doStuff() { | |
val bb = new ByteBuffer(16) | |
val p = new MutablePersonalInfoImpl(bb) | |
//these setters will store data in the bytebuffer ... | |
p.height = 5.8f | |
p.weight = 178 | |
... | |
val p2 = new PersonalInfoImpl(bb) | |
//we can read data from the bytebuffer with normal calls to the getters | |
val bmi = p2.weight * 703 / (math.pow(p2.height * 12,2)) | |
} |
You can also take a look at the unit tests or just check out the full project and run the tests yourself.
Conclusion
In these two blog posts, we’ve learned how to use scala macro annotations to automatically expand a class definition so that it implements a trait. Along the way we’ve learned the basics of working with ASTs, how to simplify our lives with quasiquotes, and how to use reflection to explore types. Our macro isn’t the most robust yet – it could use a lot more error handling – but we’ve got a taste for how everything works.
I won’t promise more in this series, but there are few more ideas I’d like to explore. First, Eugene Burmako suggested that I could achieve the functionality I want in my library using normal macros, instead of macro annotations. While it would change the user api somewhat, this is particularly appealing because macro annotations won’t make it into scala until 2.12 at the earliest.
Second, now that I’ve got a proof of concept, I’d really like to explore the idea of having classes store their data in ByteBuffer
s. Probably most readers are only interested in the discussion of using macros and reflection, but aren’t sure what the point of my macro is. I hope that by expanding these ideas somewhat, I can make it easy to store general purpose data structures directly in byte buffers. That can have all sorts of potential benefits: save memory, avoid serialization, store data off-heap, and lead to more cache-aware data structures (particularly important for numerical computing). And by using macros, we can still keep a clean user api. But, it’s still just an idea, I need to prove those claims.
I hope you’ve found this helpful in your exploration of macros. Please let me know if you found this useful, if parts are unclear, or if you found any errors.
Thanks to all my coworkers at Quantifind for all their help proofreading this!