diff --git a/LICENSE b/LICENSE index 10bee76..6a6444d 100644 --- a/LICENSE +++ b/LICENSE @@ -1,6 +1,6 @@ MIT License -Copyright (c) 2017 Chad Retz +Copyright (c) 2018 Chad Retz Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal diff --git a/README.md b/README.md index 3e7d123..f019d27 100644 --- a/README.md +++ b/README.md @@ -183,9 +183,16 @@ JVM languages. ### Getting -The latest tag can be added to your build script via [JitPack](https://jitpack.io). For example, -[here](https://jitpack.io/#cretz/asmble/0.1.0) are instructions for using the 0.1.0 release and -[here](https://jitpack.io/#cretz/asmble/master-SNAPSHOT) are instructions for the latest master. +The compiler and annotations are deployed to Maven Central. The compiler is written in Kotlin and can be added as a +Gradle dependency with: + + compile 'com.github.cretz.asmble:asmble-compiler:0.3.0' + +This is only needed to compile of course, the compiled code has no runtime requirement. The compiled code does include +some annotations (but in Java its ok to have annotations that are not found). If you do want to reflect the annotations, +the annotation library can be added as a Gradle dependency with: + + compile 'com.github.cretz.asmble:asmble-annotations:0.3.0' ### Building and Testing @@ -256,15 +263,16 @@ In the WebAssembly MVP a table is just a set of function pointers. This is store #### Globals -Globals are stored as fields on the class. A non-import global is simply a field, but an import global is a -`MethodHandle` to the getter (and would be a `MethodHandle` to the setter if mutable globals were supported). Any values -for the globals are set in the constructor. +Globals are stored as fields on the class. A non-import global is simply a field that is final if not mutable. An import +global is a `MethodHandle` to the getter and a `MethodHandle` to the setter if mutable. Any values for the globals are +set in the constructor. #### Imports The constructor accepts all imports as params. Memory is imported via a `ByteBuffer` param, then function -imports as `MethodHandle` params, then global imports as `MethodHandle` params, then a `MethodHandle` array param for an -imported table. All of these values are set as fields in the constructor. +imports as `MethodHandle` params, then global imports as `MethodHandle` params (one for getter and another for setter if +mutable), then a `MethodHandle` array param for an imported table. All of these values are set as fields in the +constructor. #### Exports @@ -363,9 +371,12 @@ stack (e.g. some places where we do a swap). Below are some performance and implementation quirks where there is a bit of an impedance mismatch between WebAssembly and the JVM: -* WebAssembly has a nice data section for byte arrays whereas the JVM does not. Right now we build a byte array from - a bunch of consts at runtime which is multiple operations per byte. This can bloat the class file size, but is quite - fast compared to alternatives such as string constants. +* WebAssembly has a nice data section for byte arrays whereas the JVM does not. Right now we use a single-byte-char + string constant (i.e. ISO-8859 charset). This saves class file size, but this means we call `String::getBytes` on + init to load bytes from the string constant. Due to the JVM using an unsigned 16-bit int as the string constant + length, the maximum byte length is 65536. Since the string constants are stored as UTF-8 constants, they can be up to + four bytes a character. Therefore, we populate memory in data chunks no larger than 16300 (nice round number to make + sure that even in the worse case of 4 bytes per char in UTF-8 view, we're still under the max). * The JVM makes no guarantees about trailing bits being preserved on NaN floating point representations like WebAssembly does. This causes some mismatch on WebAssembly tests depending on how the JVM "feels" (I haven't dug into why some bit patterns stay and some don't when NaNs are passed through methods). @@ -417,6 +428,8 @@ WASM compiled from Rust, C, Java, etc if e.g. they all have their own way of han definition of an importable set of modules that does all of these things, even if it's in WebIDL. I dunno, maybe the effort is already there, I haven't really looked. +There is https://github.com/konsoletyper/teavm + **So I can compile something in C via Emscripten and have it run on the JVM with this?** Yes, but work is required. WebAssembly is lacking any kind of standard library. So Emscripten will either embed it or diff --git a/annotations/src/main/java/asmble/annotation/WasmImport.java b/annotations/src/main/java/asmble/annotation/WasmImport.java index 437b452..ec9f77a 100644 --- a/annotations/src/main/java/asmble/annotation/WasmImport.java +++ b/annotations/src/main/java/asmble/annotation/WasmImport.java @@ -13,4 +13,5 @@ public @interface WasmImport { WasmExternalKind kind(); int resizableLimitInitial() default -1; int resizableLimitMaximum() default -1; + boolean globalSetter() default false; } diff --git a/build.gradle b/build.gradle index 31cd97f..b3bb18e 100644 --- a/build.gradle +++ b/build.gradle @@ -2,7 +2,7 @@ group 'asmble' version '0.2.0' buildscript { - ext.kotlin_version = '1.2.41' + ext.kotlin_version = '1.2.51' ext.asm_version = '5.2' repositories { @@ -19,12 +19,24 @@ buildscript { allprojects { apply plugin: 'java' + group 'com.github.cretz.asmble' + version '0.4.0-SNAPSHOT' repositories { mavenCentral() } } +project(':annotations') { + javadoc { + options.links 'https://docs.oracle.com/javase/8/docs/api/' + // TODO: change when https://github.com/gradle/gradle/issues/2354 is fixed + options.addStringOption 'Xdoclint:all', '-Xdoclint:-missing' + } + + publishSettings(project, 'asmble-annotations', 'Asmble WASM Annotations', true) +} + project(':compiler') { apply plugin: 'kotlin' apply plugin: 'application' @@ -45,6 +57,8 @@ project(':compiler') { testCompile "org.jetbrains.kotlin:kotlin-test-junit:$kotlin_version" testCompile "org.ow2.asm:asm-debug-all:$asm_version" } + + publishSettings(project, 'asmble-compiler', 'Asmble WASM Compiler', false) } project(':examples') { @@ -149,4 +163,77 @@ project(':examples:rust-string') { dependsOn compileRustWasm } mainClassName = 'asmble.examples.ruststring.Main' +} + +def publishSettings(project, projectName, projectDescription, includeJavadoc) { + project.with { + if (!project.hasProperty('ossrhUsername')) return + apply plugin: 'maven' + apply plugin: 'signing' + + archivesBaseName = projectName + + task packageSources(type: Jar) { + classifier = 'sources' + from sourceSets.main.allSource + } + + if (includeJavadoc) { + task packageJavadoc(type: Jar, dependsOn: 'javadoc') { + from javadoc.destinationDir + classifier = 'javadoc' + } + } else { + task packageJavadoc(type: Jar) { + // Empty to satisfy Sonatype's javadoc.jar requirement + classifier 'javadoc' + } + } + + artifacts { + archives packageSources, packageJavadoc + } + + signing { + sign configurations.archives + } + + uploadArchives { + repositories { + mavenDeployer { + beforeDeployment { MavenDeployment deployment -> signing.signPom(deployment) } + repository(url: 'https://oss.sonatype.org/service/local/staging/deploy/maven2/') { + authentication(userName: ossrhUsername, password: ossrhPassword) + } + snapshotRepository(url: 'https://oss.sonatype.org/content/repositories/snapshots/') { + authentication(userName: ossrhUsername, password: ossrhPassword) + } + pom.project { + name projectName + packaging 'jar' + description projectDescription + url 'https://github.com/cretz/asmble' + scm { + connection 'scm:git:git@github.com:cretz/asmble.git' + developerConnection 'scm:git:git@github.com:cretz/asmble.git' + url 'git@github.com:cretz/asmble.git' + } + licenses { + license { + name 'MIT License' + url 'https://opensource.org/licenses/MIT' + } + } + developers { + developer { + id 'cretz' + name 'Chad Retz' + url 'https://github.com/cretz' + } + } + } + } + } + } + } } \ No newline at end of file diff --git a/compiler/src/main/kotlin/asmble/ast/Node.kt b/compiler/src/main/kotlin/asmble/ast/Node.kt index 2fb2120..c8e4608 100644 --- a/compiler/src/main/kotlin/asmble/ast/Node.kt +++ b/compiler/src/main/kotlin/asmble/ast/Node.kt @@ -27,6 +27,7 @@ sealed class Node { val elems: List = emptyList(), val funcs: List = emptyList(), val data: List = emptyList(), + val names: NameSection? = null, val customSections: List = emptyList() ) : Node() @@ -160,6 +161,12 @@ sealed class Node { } } + data class NameSection( + val moduleName: String?, + val funcNames: Map, + val localNames: Map> + ) : Node() + sealed class Instr : Node() { fun op() = InstrOp.classToOpMap[this::class] ?: throw Exception("No op found for ${this::class}") diff --git a/compiler/src/main/kotlin/asmble/cli/Compile.kt b/compiler/src/main/kotlin/asmble/cli/Compile.kt index cd3bceb..b0b3ea6 100644 --- a/compiler/src/main/kotlin/asmble/cli/Compile.kt +++ b/compiler/src/main/kotlin/asmble/cli/Compile.kt @@ -51,7 +51,7 @@ open class Compile : Command() { val inFormat = if (args.inFormat != "") args.inFormat else args.inFile.substringAfterLast('.', "") - val script = Translate.inToAst(args.inFile, inFormat) + val script = Translate().also { it.logger = logger }.inToAst(args.inFile, inFormat) val mod = (script.commands.firstOrNull() as? Script.Cmd.Module) ?: error("Only a single sexpr for (module) allowed") val outStream = when (args.outFile) { diff --git a/compiler/src/main/kotlin/asmble/cli/Translate.kt b/compiler/src/main/kotlin/asmble/cli/Translate.kt index d2ff48f..5d0ef7e 100644 --- a/compiler/src/main/kotlin/asmble/cli/Translate.kt +++ b/compiler/src/main/kotlin/asmble/cli/Translate.kt @@ -87,7 +87,7 @@ open class Translate : Command() { } } "wasm" -> - Script(listOf(Script.Cmd.Module(BinaryToAst.toModule( + Script(listOf(Script.Cmd.Module(BinaryToAst(logger = logger).toModule( ByteReader.InputStream(inBytes.inputStream())), null))) else -> error("Unknown in format '$inFormat'") } diff --git a/compiler/src/main/kotlin/asmble/compile/jvm/AsmExt.kt b/compiler/src/main/kotlin/asmble/compile/jvm/AsmExt.kt index d17385a..3c3586a 100644 --- a/compiler/src/main/kotlin/asmble/compile/jvm/AsmExt.kt +++ b/compiler/src/main/kotlin/asmble/compile/jvm/AsmExt.kt @@ -210,3 +210,7 @@ fun ByteArray.asClassNode(): ClassNode { ClassReader(this).accept(newNode, 0) return newNode } + +fun ByteArray.chunked(v: Int) = (0 until size step v).asSequence().map { + copyOfRange(it, (it + v).takeIf { it < size } ?: size) +} \ No newline at end of file diff --git a/compiler/src/main/kotlin/asmble/compile/jvm/AstToAsm.kt b/compiler/src/main/kotlin/asmble/compile/jvm/AstToAsm.kt index 0e53ac2..4552a47 100644 --- a/compiler/src/main/kotlin/asmble/compile/jvm/AstToAsm.kt +++ b/compiler/src/main/kotlin/asmble/compile/jvm/AstToAsm.kt @@ -48,10 +48,13 @@ open class AstToAsm { }) // Now all import globals as getter (and maybe setter) method handles ctx.cls.fields.addAll(ctx.importGlobals.mapIndexed { index, import -> - if ((import.kind as Node.Import.Kind.Global).type.mutable) throw CompileErr.MutableGlobalImport(index) - FieldNode(Opcodes.ACC_PRIVATE + Opcodes.ACC_FINAL, ctx.importGlobalGetterFieldName(index), + val getter = FieldNode(Opcodes.ACC_PRIVATE + Opcodes.ACC_FINAL, ctx.importGlobalGetterFieldName(index), MethodHandle::class.ref.asmDesc, null, null) - }) + if (!(import.kind as Node.Import.Kind.Global).type.mutable) listOf(getter) + else listOf(getter, FieldNode( + Opcodes.ACC_PRIVATE + Opcodes.ACC_FINAL, ctx.importGlobalSetterFieldName(index), + MethodHandle::class.ref.asmDesc, null, null)) + }.flatten()) // Now all non-import globals ctx.cls.fields.addAll(ctx.mod.globals.mapIndexed { index, global -> val access = Opcodes.ACC_PRIVATE + if (!global.type.mutable) Opcodes.ACC_FINAL else 0 @@ -181,9 +184,11 @@ open class AstToAsm { fun constructorImportTypes(ctx: ClsContext) = ctx.importFuncs.map { MethodHandle::class.ref } + - // We know it's only getters - ctx.importGlobals.map { MethodHandle::class.ref } + - ctx.mod.imports.filter { it.kind is Node.Import.Kind.Table }.map { Array::class.ref } + ctx.importGlobals.flatMap { + // If it's mutable, it also comes with a setter + if ((it.kind as? Node.Import.Kind.Global)?.type?.mutable == false) listOf(MethodHandle::class.ref) + else listOf(MethodHandle::class.ref, MethodHandle::class.ref) + } + ctx.mod.imports.filter { it.kind is Node.Import.Kind.Table }.map { Array::class.ref } fun toConstructorNode(ctx: ClsContext, func: Func) = mutableListOf>().let { paramAnns -> // If the first param is a mem class and imported, add annotation @@ -200,7 +205,15 @@ open class AstToAsm { } // All non-mem imports one after another ctx.importFuncs.forEach { paramAnns.add(listOf(importAnnotation(ctx, it))) } - ctx.importGlobals.forEach { paramAnns.add(listOf(importAnnotation(ctx, it))) } + ctx.importGlobals.forEach { + paramAnns.add(listOf(importAnnotation(ctx, it))) + // There are two annotations here if it's mutable + if ((it.kind as? Node.Import.Kind.Global)?.type?.mutable == true) + paramAnns.add(listOf(importAnnotation(ctx, it).also { + it.values.add("globalSetter") + it.values.add(true) + })) + } ctx.mod.imports.forEach { if (it.kind is Node.Import.Kind.Table) paramAnns.add(listOf(importAnnotation(ctx, it))) } @@ -241,14 +254,25 @@ open class AstToAsm { } fun setConstructorGlobalImports(ctx: ClsContext, func: Func, paramsBeforeImports: Int) = - ctx.importGlobals.indices.fold(func) { func, importIndex -> + ctx.importGlobals.foldIndexed(func to ctx.importFuncs.size + paramsBeforeImports) { + importIndex, (func, importParamOffset), import -> + // Always a getter handle func.addInsns( VarInsnNode(Opcodes.ALOAD, 0), - VarInsnNode(Opcodes.ALOAD, ctx.importFuncs.size + importIndex + paramsBeforeImports + 1), + VarInsnNode(Opcodes.ALOAD, importParamOffset + 1), FieldInsnNode(Opcodes.PUTFIELD, ctx.thisRef.asmName, ctx.importGlobalGetterFieldName(importIndex), MethodHandle::class.ref.asmDesc) - ) - } + ).let { func -> + // If it's mutable, it has a second setter handle + if ((import.kind as? Node.Import.Kind.Global)?.type?.mutable == false) func to importParamOffset + 1 + else func.addInsns( + VarInsnNode(Opcodes.ALOAD, 0), + VarInsnNode(Opcodes.ALOAD, importParamOffset + 2), + FieldInsnNode(Opcodes.PUTFIELD, ctx.thisRef.asmName, + ctx.importGlobalSetterFieldName(importIndex), MethodHandle::class.ref.asmDesc) + ) to importParamOffset + 2 + } + }.first fun setConstructorFunctionImports(ctx: ClsContext, func: Func, paramsBeforeImports: Int) = ctx.importFuncs.indices.fold(func) { func, importIndex -> @@ -262,7 +286,10 @@ open class AstToAsm { fun setConstructorTableImports(ctx: ClsContext, func: Func, paramsBeforeImports: Int) = if (ctx.mod.imports.none { it.kind is Node.Import.Kind.Table }) func else { - val importIndex = ctx.importFuncs.size + ctx.importGlobals.size + paramsBeforeImports + 1 + val importIndex = ctx.importFuncs.size + + // Mutable global imports have setters and take up two spots + ctx.importGlobals.sumBy { if ((it.kind as? Node.Import.Kind.Global)?.type?.mutable == true) 2 else 1 } + + paramsBeforeImports + 1 func.addInsns( VarInsnNode(Opcodes.ALOAD, 0), VarInsnNode(Opcodes.ALOAD, importIndex), @@ -300,11 +327,14 @@ open class AstToAsm { global.type.contentType.typeRef, refGlobalKind.type.contentType.typeRef ) + val paramOffset = ctx.importFuncs.size + paramsBeforeImports + 1 + + ctx.importGlobals.take(it.index).sumBy { + // Immutable jumps 1, mutable jumps 2 + if ((it.kind as? Node.Import.Kind.Global)?.type?.mutable == false) 1 + else 2 + } listOf( - VarInsnNode( - Opcodes.ALOAD, - ctx.importFuncs.size + it.index + paramsBeforeImports + 1 - ), + VarInsnNode(Opcodes.ALOAD, paramOffset), MethodInsnNode( Opcodes.INVOKEVIRTUAL, MethodHandle::class.ref.asmName, @@ -357,7 +387,10 @@ open class AstToAsm { // Otherwise, it was imported and we can set the elems on the imported one // from the parameter // TODO: I think this is a security concern and bad practice, may revisit - val importIndex = ctx.importFuncs.size + ctx.importGlobals.size + paramsBeforeImports + 1 + val importIndex = ctx.importFuncs.size + ctx.importGlobals.sumBy { + // Immutable is 1, mutable is 2 + if ((it.kind as? Node.Import.Kind.Global)?.type?.mutable == false) 1 else 2 + } + paramsBeforeImports + 1 return func.addInsns(VarInsnNode(Opcodes.ALOAD, importIndex)). let { func -> addElemsToTable(ctx, func, paramsBeforeImports) }. // Remove the array that's still there @@ -533,28 +566,58 @@ open class AstToAsm { is Either.Left -> (global.v.kind as Node.Import.Kind.Global).type is Either.Right -> global.v.type } - if (type.mutable) throw CompileErr.MutableGlobalExport(export.index) // Create a simple getter - val method = MethodNode(Opcodes.ACC_PUBLIC, "get" + export.field.javaIdent.capitalize(), + val getter = MethodNode(Opcodes.ACC_PUBLIC, "get" + export.field.javaIdent.capitalize(), "()" + type.contentType.typeRef.asmDesc, null, null) - method.addInsns(VarInsnNode(Opcodes.ALOAD, 0)) - if (global is Either.Left) method.addInsns( + getter.addInsns(VarInsnNode(Opcodes.ALOAD, 0)) + if (global is Either.Left) getter.addInsns( FieldInsnNode(Opcodes.GETFIELD, ctx.thisRef.asmName, ctx.importGlobalGetterFieldName(export.index), MethodHandle::class.ref.asmDesc), MethodInsnNode(Opcodes.INVOKEVIRTUAL, MethodHandle::class.ref.asmName, "invokeExact", "()" + type.contentType.typeRef.asmDesc, false) - ) else method.addInsns( + ) else getter.addInsns( FieldInsnNode(Opcodes.GETFIELD, ctx.thisRef.asmName, ctx.globalName(export.index), type.contentType.typeRef.asmDesc) ) - method.addInsns(InsnNode(when (type.contentType) { + getter.addInsns(InsnNode(when (type.contentType) { Node.Type.Value.I32 -> Opcodes.IRETURN Node.Type.Value.I64 -> Opcodes.LRETURN Node.Type.Value.F32 -> Opcodes.FRETURN Node.Type.Value.F64 -> Opcodes.DRETURN })) - method.visibleAnnotations = listOf(exportAnnotation(export)) - ctx.cls.methods.plusAssign(method) + getter.visibleAnnotations = listOf(exportAnnotation(export)) + ctx.cls.methods.plusAssign(getter) + // If mutable, create simple setter + if (type.mutable) { + val setter = MethodNode(Opcodes.ACC_PUBLIC, "set" + export.field.javaIdent.capitalize(), + "(${type.contentType.typeRef.asmDesc})V", null, null) + setter.addInsns(VarInsnNode(Opcodes.ALOAD, 0)) + if (global is Either.Left) setter.addInsns( + FieldInsnNode(Opcodes.GETFIELD, ctx.thisRef.asmName, + ctx.importGlobalSetterFieldName(export.index), MethodHandle::class.ref.asmDesc), + VarInsnNode(when (type.contentType) { + Node.Type.Value.I32 -> Opcodes.ILOAD + Node.Type.Value.I64 -> Opcodes.LLOAD + Node.Type.Value.F32 -> Opcodes.FLOAD + Node.Type.Value.F64 -> Opcodes.DLOAD + }, 1), + MethodInsnNode(Opcodes.INVOKEVIRTUAL, MethodHandle::class.ref.asmName, "invokeExact", + "(${type.contentType.typeRef.asmDesc})V", false), + InsnNode(Opcodes.RETURN) + ) else setter.addInsns( + VarInsnNode(when (type.contentType) { + Node.Type.Value.I32 -> Opcodes.ILOAD + Node.Type.Value.I64 -> Opcodes.LLOAD + Node.Type.Value.F32 -> Opcodes.FLOAD + Node.Type.Value.F64 -> Opcodes.DLOAD + }, 1), + FieldInsnNode(Opcodes.PUTFIELD, ctx.thisRef.asmName, ctx.globalName(export.index), + type.contentType.typeRef.asmDesc), + InsnNode(Opcodes.RETURN) + ) + setter.visibleAnnotations = listOf(exportAnnotation(export)) + ctx.cls.methods.plusAssign(setter) + } } fun addExportMemory(ctx: ClsContext, export: Node.Export) { diff --git a/compiler/src/main/kotlin/asmble/compile/jvm/ByteBufferMem.kt b/compiler/src/main/kotlin/asmble/compile/jvm/ByteBufferMem.kt index ec04248..9a1c3ba 100644 --- a/compiler/src/main/kotlin/asmble/compile/jvm/ByteBufferMem.kt +++ b/compiler/src/main/kotlin/asmble/compile/jvm/ByteBufferMem.kt @@ -46,16 +46,25 @@ open class ByteBufferMem(val direct: Boolean = true) : Mem { let(buildOffset).popExpecting(Int::class.ref). addInsns( forceFnType Buffer>(ByteBuffer::position).invokeVirtual(), - TypeInsnNode(Opcodes.CHECKCAST, memType.asmName), - // We're going to do this as an LDC string in ISO-8859 and read it back at runtime - LdcInsnNode(bytes.toString(Charsets.ISO_8859_1)), - LdcInsnNode("ISO-8859-1"), - // Ug, can't do func refs on native types here... - MethodInsnNode(Opcodes.INVOKEVIRTUAL, String::class.ref.asmName, - "getBytes", "(Ljava/lang/String;)[B", false), - 0.const, - bytes.size.const, - forceFnType ByteBuffer>(ByteBuffer::put).invokeVirtual(), + TypeInsnNode(Opcodes.CHECKCAST, memType.asmName) + ).addInsns( + // We're going to do this as an LDC string in ISO-8859 and read it back at runtime. However, + // due to JVM limits, we can't have a string > 65536 chars. We chunk into 16300 because when + // converting to UTF8 const it can be up to 4 bytes per char, so this makes sure it doesn't + // overflow. + bytes.chunked(16300).flatMap { bytes -> + sequenceOf( + LdcInsnNode(bytes.toString(Charsets.ISO_8859_1)), + LdcInsnNode("ISO-8859-1"), + // Ug, can't do func refs on native types here... + MethodInsnNode(Opcodes.INVOKEVIRTUAL, String::class.ref.asmName, + "getBytes", "(Ljava/lang/String;)[B", false), + 0.const, + bytes.size.const, + forceFnType ByteBuffer>(ByteBuffer::put).invokeVirtual() + ) + }.toList() + ).addInsns( InsnNode(Opcodes.POP) ) diff --git a/compiler/src/main/kotlin/asmble/compile/jvm/ClsContext.kt b/compiler/src/main/kotlin/asmble/compile/jvm/ClsContext.kt index 0827825..8fecdca 100644 --- a/compiler/src/main/kotlin/asmble/compile/jvm/ClsContext.kt +++ b/compiler/src/main/kotlin/asmble/compile/jvm/ClsContext.kt @@ -39,6 +39,24 @@ data class ClsContext( val hasTable: Boolean by lazy { mod.tables.isNotEmpty() || mod.imports.any { it.kind is Node.Import.Kind.Table } } + val dedupedFuncNames: Map? by lazy { + // Consider all exports as seen + val seen = mod.exports.flatMap { export -> + when { + export.kind == Node.ExternalKind.FUNCTION -> listOf(export.field.javaIdent) + // Just to make it easy, consider all globals as having setters + export.kind == Node.ExternalKind.GLOBAL -> + export.field.javaIdent.capitalize().let { listOf("get$it", "set$it") } + else -> listOf("get" + export.field.javaIdent.capitalize()) + } + }.toMutableSet() + mod.names?.funcNames?.toList()?.sortedBy { it.first }?.map { (index, origName) -> + var name = origName.javaIdent + var nameIndex = 0 + while (!seen.add(name)) name = origName.javaIdent + (nameIndex++) + index to name + }?.toMap() + } fun assertHasMemory() { if (!hasMemory) throw CompileErr.UnknownMemory(0) } @@ -71,7 +89,7 @@ data class ClsContext( fun importGlobalGetterFieldName(index: Int) = "import\$get" + globalName(index) fun importGlobalSetterFieldName(index: Int) = "import\$set" + globalName(index) fun globalName(index: Int) = "\$global$index" - fun funcName(index: Int) = "\$func$index" + fun funcName(index: Int) = dedupedFuncNames?.get(index) ?: "\$func$index" private fun syntheticFunc( nameSuffix: String, diff --git a/compiler/src/main/kotlin/asmble/compile/jvm/CompileErr.kt b/compiler/src/main/kotlin/asmble/compile/jvm/CompileErr.kt index a040f08..e5eb3a4 100644 --- a/compiler/src/main/kotlin/asmble/compile/jvm/CompileErr.kt +++ b/compiler/src/main/kotlin/asmble/compile/jvm/CompileErr.kt @@ -102,18 +102,6 @@ sealed class CompileErr(message: String, cause: Throwable? = null) : RuntimeExce override val asmErrString get() = "global is immutable" } - class MutableGlobalImport( - val index: Int - ) : CompileErr("Attempted to import mutable global at index $index") { - override val asmErrString get() = "mutable globals cannot be imported" - } - - class MutableGlobalExport( - val index: Int - ) : CompileErr("Attempted to export global $index which is mutable") { - override val asmErrString get() = "mutable globals cannot be exported" - } - class GlobalInitNotConstant( val index: Int ) : CompileErr("Expected init for global $index to be single constant value") { diff --git a/compiler/src/main/kotlin/asmble/compile/jvm/InsnReworker.kt b/compiler/src/main/kotlin/asmble/compile/jvm/InsnReworker.kt index f4f090e..c788d99 100644 --- a/compiler/src/main/kotlin/asmble/compile/jvm/InsnReworker.kt +++ b/compiler/src/main/kotlin/asmble/compile/jvm/InsnReworker.kt @@ -183,6 +183,7 @@ open class InsnReworker { if (!foundUnconditionalJump) throw CompileErr.StackInjectionMismatch(count, insn) } + var traceStackSize = 0 // Used only for trace // Go over each insn, determining where to inject insns.forEachIndexed { index, insn -> // Handle special injection cases @@ -221,8 +222,15 @@ open class InsnReworker { else -> { } } + // Log some trace output + ctx.trace { + insnStackDiff(ctx, insn).let { + traceStackSize += it + "Stack diff is $it for insn #$index $insn, stack size now: $traceStackSize" + } + } + // Add the current diff - ctx.trace { "Stack diff is ${insnStackDiff(ctx, insn)} for insn #$index $insn" } stackManips += insnStackDiff(ctx, insn) to index } @@ -266,7 +274,7 @@ open class InsnReworker { is Node.Instr.I64Load32S, is Node.Instr.I64Load32U -> POP_PARAM + PUSH_RESULT is Node.Instr.I32Store, is Node.Instr.I64Store, is Node.Instr.F32Store, is Node.Instr.F64Store, is Node.Instr.I32Store8, is Node.Instr.I32Store16, is Node.Instr.I64Store8, is Node.Instr.I64Store16, - is Node.Instr.I64Store32 -> POP_PARAM + is Node.Instr.I64Store32 -> POP_PARAM + POP_PARAM is Node.Instr.MemorySize -> PUSH_RESULT is Node.Instr.MemoryGrow -> POP_PARAM + PUSH_RESULT is Node.Instr.I32Const, is Node.Instr.I64Const, diff --git a/compiler/src/main/kotlin/asmble/io/AstToBinary.kt b/compiler/src/main/kotlin/asmble/io/AstToBinary.kt index 5272c9f..560d728 100644 --- a/compiler/src/main/kotlin/asmble/io/AstToBinary.kt +++ b/compiler/src/main/kotlin/asmble/io/AstToBinary.kt @@ -5,6 +5,7 @@ import asmble.util.toRawIntBits import asmble.util.toRawLongBits import asmble.util.toUnsignedBigInt import asmble.util.toUnsignedLong +import java.io.ByteArrayOutputStream open class AstToBinary(val version: Long = 1L) { @@ -140,6 +141,9 @@ open class AstToBinary(val version: Long = 1L) { fromResizableLimits(b, n.limits) } + fun fromModule(n: Node.Module) = + ByteArrayOutputStream().also { fromModule(ByteWriter.OutputStream(it), n) }.toByteArray() + fun fromModule(b: ByteWriter, n: Node.Module) { b.writeUInt32(0x6d736100) b.writeUInt32(version) @@ -160,10 +164,33 @@ open class AstToBinary(val version: Long = 1L) { wrapListSection(b, n, 9, n.elems, this::fromElem) wrapListSection(b, n, 10, n.funcs, this::fromFuncBody) wrapListSection(b, n, 11, n.data, this::fromData) + n.names?.also { fromNames(b, it) } // All other custom sections after the previous n.customSections.filter { it.afterSectionId > 11 }.forEach { fromCustomSection(b, it) } } + fun fromNames(b: ByteWriter, n: Node.NameSection) { + fun indexMap(b: ByteWriter, map: Map, fn: (T) -> Unit) { + b.writeVarUInt32(map.size) + map.forEach { index, v -> b.writeVarUInt32(index).also { fn(v) } } + } + fun nameMap(b: ByteWriter, map: Map) = indexMap(b, map) { b.writeString(it) } + b.writeVarUInt7(0) + b.withVarUInt32PayloadSizePrepended { b -> + b.writeString("name") + n.moduleName?.also { moduleName -> + b.writeVarUInt7(0) + b.withVarUInt32PayloadSizePrepended { b -> b.writeString(moduleName) } + } + if (n.funcNames.isNotEmpty()) b.writeVarUInt7(1).also { + b.withVarUInt32PayloadSizePrepended { b -> nameMap(b, n.funcNames) } + } + if (n.localNames.isNotEmpty()) b.writeVarUInt7(2).also { + b.withVarUInt32PayloadSizePrepended { b -> indexMap(b, n.localNames) { nameMap(b, it) } } + } + } + } + fun fromResizableLimits(b: ByteWriter, n: Node.ResizableLimits) { b.writeVarUInt1(n.maximum != null) b.writeVarUInt32(n.initial) diff --git a/compiler/src/main/kotlin/asmble/io/AstToSExpr.kt b/compiler/src/main/kotlin/asmble/io/AstToSExpr.kt index 199ae9c..ef83955 100644 --- a/compiler/src/main/kotlin/asmble/io/AstToSExpr.kt +++ b/compiler/src/main/kotlin/asmble/io/AstToSExpr.kt @@ -62,13 +62,21 @@ open class AstToSExpr(val parensInstrs: Boolean = true) { Node.ExternalKind.GLOBAL -> newMulti("global") + v.index } - fun fromFunc(v: Node.Func, name: String? = null, impExp: ImportOrExport? = null) = - newMulti("func", name) + impExp?.let(this::fromImportOrExport) + fromFuncSig(v.type) + - fromLocals(v.locals) + fromInstrs(v.instructions).unwrapInstrs() + fun fromFunc( + v: Node.Func, + name: String? = null, + impExp: ImportOrExport? = null, + localNames: Map = emptyMap() + ) = + newMulti("func", name) + impExp?.let(this::fromImportOrExport) + fromFuncSig(v.type, localNames) + + fromLocals(v.locals, v.type.params.size, localNames) + fromInstrs(v.instructions).unwrapInstrs() - fun fromFuncSig(v: Node.Type.Func): List { + fun fromFuncSig(v: Node.Type.Func, localNames: Map = emptyMap()): List { var ret = emptyList() - if (v.params.isNotEmpty()) ret += newMulti("param") + v.params.map(this::fromType) + if (v.params.isNotEmpty()) { + if (localNames.isEmpty()) ret += newMulti("param") + v.params.map(this::fromType) + else ret += v.params.mapIndexed { index, param -> newMulti("param", localNames[index]) + fromType(param) } + } v.ret?.also { ret += newMulti("result") + fromType(it) } return ret } @@ -80,8 +88,8 @@ open class AstToSExpr(val parensInstrs: Boolean = true) { fun fromGlobalSig(v: Node.Type.Global) = if (v.mutable) newMulti("mut") + fromType(v.contentType) else fromType(v.contentType) - fun fromImport(v: Node.Import, types: List) = - (newMulti("import") + v.module.quoted) + v.field.quoted + fromImportKind(v.kind, types) + fun fromImport(v: Node.Import, types: List, name: String? = null) = + (newMulti("import") + v.module.quoted) + v.field.quoted + fromImportKind(v.kind, types, name) fun fromImportFunc(v: Node.Import.Kind.Func, types: List, name: String? = null) = fromImportFunc(types.getOrElse(v.typeIndex) { throw Exception("No type at ${v.typeIndex}") }, name) @@ -91,11 +99,11 @@ open class AstToSExpr(val parensInstrs: Boolean = true) { fun fromImportGlobal(v: Node.Import.Kind.Global, name: String? = null) = newMulti("global", name) + fromGlobalSig(v.type) - fun fromImportKind(v: Node.Import.Kind, types: List) = when(v) { - is Node.Import.Kind.Func -> fromImportFunc(v, types) - is Node.Import.Kind.Table -> fromImportTable(v) - is Node.Import.Kind.Memory -> fromImportMemory(v) - is Node.Import.Kind.Global -> fromImportGlobal(v) + fun fromImportKind(v: Node.Import.Kind, types: List, name: String? = null) = when(v) { + is Node.Import.Kind.Func -> fromImportFunc(v, types, name) + is Node.Import.Kind.Table -> fromImportTable(v, name) + is Node.Import.Kind.Memory -> fromImportMemory(v, name) + is Node.Import.Kind.Global -> fromImportGlobal(v, name) } fun fromImportMemory(v: Node.Import.Kind.Memory, name: String? = null) = @@ -161,8 +169,10 @@ open class AstToSExpr(val parensInstrs: Boolean = true) { return listOf(SExpr.Multi(untilNext().first)) } - fun fromLocals(v: List) = - if (v.isEmpty()) null else newMulti("local") + v.map(this::fromType) + fun fromLocals(v: List, paramOffset: Int, localNames: Map = emptyMap()) = + if (v.isEmpty()) emptyList() + else if (localNames.isEmpty()) listOf(newMulti("local") + v.map(this::fromType)) + else v.mapIndexed { index, v -> newMulti("local", localNames[paramOffset + index]) + fromType(v) } fun fromMemory(v: Node.Type.Memory, name: String? = null, impExp: ImportOrExport? = null) = newMulti("memory", name) + impExp?.let(this::fromImportOrExport) + fromMemorySig(v) @@ -175,7 +185,7 @@ open class AstToSExpr(val parensInstrs: Boolean = true) { is Script.Cmd.Meta.Output -> newMulti("output", v.name) + v.str } - fun fromModule(v: Node.Module, name: String? = null): SExpr.Multi { + fun fromModule(v: Node.Module, name: String? = v.names?.moduleName): SExpr.Multi { var ret = newMulti("module", name) // If there is a call_indirect, then we need to output all types in exact order. @@ -187,8 +197,14 @@ open class AstToSExpr(val parensInstrs: Boolean = true) { v.types.filterIndexed { i, _ -> importIndices.contains(i) } - v.funcs.map { it.type } } + // Keep track of the current function index for names + var funcIndex = -1 + ret += types.map { fromTypeDef(it) } - ret += v.imports.map { fromImport(it, v.types) } + ret += v.imports.map { + if (it.kind is Node.Import.Kind.Func) funcIndex++ + fromImport(it, v.types, v.names?.funcNames?.get(funcIndex)) + } ret += v.exports.map(this::fromExport) ret += v.tables.map { fromTable(it) } ret += v.memories.map { fromMemory(it) } @@ -196,7 +212,14 @@ open class AstToSExpr(val parensInstrs: Boolean = true) { ret += v.elems.map(this::fromElem) ret += v.data.map(this::fromData) ret += v.startFuncIndex?.let(this::fromStart) - ret += v.funcs.map { fromFunc(it) } + ret += v.funcs.map { + funcIndex++ + fromFunc( + v = it, + name = v.names?.funcNames?.get(funcIndex), + localNames = v.names?.localNames?.get(funcIndex) ?: emptyMap() + ) + } return ret } @@ -235,10 +258,8 @@ open class AstToSExpr(val parensInstrs: Boolean = true) { if (exp == null) this else this.copy(vals = this.vals + exp) private operator fun SExpr.Multi.plus(exps: List?) = if (exps == null || exps.isEmpty()) this else this.copy(vals = this.vals + exps) - private fun newMulti(initSymb: String? = null, initName: String? = null): SExpr.Multi { - initName?.also { require(it.startsWith("$")) } - return SExpr.Multi() + initSymb + initName - } + private fun newMulti(initSymb: String? = null, initName: String? = null) = + SExpr.Multi() + initSymb + initName?.let { "$$it" } private fun List.unwrapInstrs() = if (parensInstrs) this else this.single().vals private val String.quoted get() = fromString(this, true) diff --git a/compiler/src/main/kotlin/asmble/io/BinaryToAst.kt b/compiler/src/main/kotlin/asmble/io/BinaryToAst.kt index 1a88b94..67db443 100644 --- a/compiler/src/main/kotlin/asmble/io/BinaryToAst.kt +++ b/compiler/src/main/kotlin/asmble/io/BinaryToAst.kt @@ -2,11 +2,13 @@ package asmble.io import asmble.ast.Node import asmble.util.* +import java.io.ByteArrayInputStream import java.nio.ByteBuffer open class BinaryToAst( val version: Long = 1L, - val logger: Logger = Logger.Print(Logger.Level.OFF) + val logger: Logger = Logger.Print(Logger.Level.OFF), + val includeNameSection: Boolean = true ) : Logger by logger { fun toBlockType(b: ByteReader) = b.readVarInt7().toInt().let { @@ -19,6 +21,23 @@ open class BinaryToAst( payload = b.readBytes() ) + fun toNameSection(b: ByteReader) = generateSequence { + if (b.isEof) null + else b.readVarUInt7().toInt() to b.read(b.readVarUInt32AsInt()) + }.fold(Node.NameSection(null, emptyMap(), emptyMap())) { sect, (type, b) -> + fun indexMap(b: ByteReader, fn: (ByteReader) -> T) = + b.readList { it.readVarUInt32AsInt() to fn(it) }.let { pairs -> + pairs.toMap().also { require(it.size == pairs.size) { "Malformed names: duplicate indices" } } + } + fun nameMap(b: ByteReader) = indexMap(b) { it.readString() } + when (type) { + 0 -> sect.copy(moduleName = b.readString()) + 1 -> sect.copy(funcNames = nameMap(b)) + 2 -> sect.copy(localNames = indexMap(b, ::nameMap)) + else -> error("Malformed names: unrecognized type: $type") + }.also { require(b.isEof) } + } + fun toData(b: ByteReader) = Node.Data( index = b.readVarUInt32AsInt(), offset = toInitExpr(b), @@ -144,28 +163,31 @@ open class BinaryToAst( } } - fun toLocals(b: ByteReader) = b.readVarUInt32AsInt().let { size -> - toValueType(b).let { type -> List(size) { type } } + fun toLocals(b: ByteReader): List { + val size = try { b.readVarUInt32AsInt() } catch (e: NumberFormatException) { throw IoErr.InvalidLocalSize(e) } + return toValueType(b).let { type -> List(size) { type } } } fun toMemoryType(b: ByteReader) = Node.Type.Memory(toResizableLimits(b)) - fun toModule(bytes: ByteReader): Node.Module { - if (bytes.readUInt32() != 0x6d736100L) throw IoErr.InvalidMagicNumber() - bytes.readUInt32().let { if (it != version) throw IoErr.InvalidVersion(it, listOf(version)) } + fun toModule(b: ByteArray) = toModule(ByteReader.InputStream(b.inputStream())) + + fun toModule(b: ByteReader): Node.Module { + if (b.readUInt32() != 0x6d736100L) throw IoErr.InvalidMagicNumber() + b.readUInt32().let { if (it != version) throw IoErr.InvalidVersion(it, listOf(version)) } // Slice up all the sections var maxSectionId = 0 var sections = emptyList>() - while (!bytes.isEof) { - val sectionId = bytes.readVarUInt7().toInt() + while (!b.isEof) { + val sectionId = b.readVarUInt7().toInt() if (sectionId > 11) throw IoErr.InvalidSectionId(sectionId) if (sectionId != 0) require(sectionId > maxSectionId) { "Section ID $sectionId came after $maxSectionId" }. also { maxSectionId = sectionId } - val sectionLen = bytes.readVarUInt32AsInt() + val sectionLen = b.readVarUInt32AsInt() // each 'read' invocation creates new InputStream and don't closes it - sections += sectionId to bytes.read(sectionLen) + sections += sectionId to b.read(sectionLen) } // Now build the module @@ -174,6 +196,7 @@ open class BinaryToAst( val types = readSectionList(1, this::toFuncType) val funcIndices = readSectionList(3) { it.readVarUInt32AsInt() } + var nameSection: Node.NameSection? = null return Node.Module( types = types, imports = readSectionList(2, this::toImport), @@ -194,10 +217,18 @@ open class BinaryToAst( val afterSectionId = if (index == 0) 0 else sections[index - 1].let { (prevSectionId, _) -> if (prevSectionId == 0) customSections.last().afterSectionId else prevSectionId } - customSections + toCustomSection(b, afterSectionId) + // Try to parse the name section + val section = toCustomSection(b, afterSectionId).takeIf { section -> + val shouldParseNames = includeNameSection && nameSection == null && section.name == "name" + !shouldParseNames || try { + nameSection = toNameSection(ByteReader.InputStream(section.payload.inputStream())) + false + } catch (e: Exception) { warn { "Failed parsing name section: $e" }; true } + } + if (section == null) customSections else customSections + section } } - ) + ).copy(names = nameSection) } fun toResizableLimits(b: ByteReader) = b.readVarUInt1().let { diff --git a/compiler/src/main/kotlin/asmble/io/IoErr.kt b/compiler/src/main/kotlin/asmble/io/IoErr.kt index 0ad7b78..e4fcb38 100644 --- a/compiler/src/main/kotlin/asmble/io/IoErr.kt +++ b/compiler/src/main/kotlin/asmble/io/IoErr.kt @@ -123,4 +123,8 @@ sealed class IoErr(message: String, cause: Throwable? = null) : RuntimeException override val asmErrString get() = "integer representation too long" override val asmErrStrings get() = listOf(asmErrString, "integer too large") } + + class InvalidLocalSize(cause: NumberFormatException) : IoErr("Invalid local size", cause) { + override val asmErrString get() = "too many locals" + } } \ No newline at end of file diff --git a/compiler/src/main/kotlin/asmble/io/SExprToAst.kt b/compiler/src/main/kotlin/asmble/io/SExprToAst.kt index 1e56683..0dd92b9 100644 --- a/compiler/src/main/kotlin/asmble/io/SExprToAst.kt +++ b/compiler/src/main/kotlin/asmble/io/SExprToAst.kt @@ -9,22 +9,27 @@ import asmble.util.* import java.io.ByteArrayInputStream import java.math.BigInteger -typealias NameMap = Map - -open class SExprToAst { +open class SExprToAst( + val includeNames: Boolean = true +) { data class ExprContext( val nameMap: NameMap, val blockDepth: Int = 0, val types: List = emptyList(), val callIndirectNeverBeforeSeenFuncTypes: MutableList = mutableListOf() - ) + ) { + companion object { + val empty = ExprContext(NameMap(emptyMap(), null, null)) + } + } data class FuncResult( val name: String?, val func: Node.Func, val importOrExport: ImportOrExport?, // These come from call_indirect insns - val additionalFuncTypesToAdd: List + val additionalFuncTypesToAdd: List, + val nameMap: NameMap ) fun toAction(exp: SExpr.Multi): Script.Cmd.Action { @@ -36,7 +41,7 @@ open class SExprToAst { return when(exp.vals.first().symbolStr()) { "invoke" -> Script.Cmd.Action.Invoke(name, str, exp.vals.drop(index).map { - toExprMaybe(it as SExpr.Multi, ExprContext(emptyMap())) + toExprMaybe(it as SExpr.Multi, ExprContext.empty) }) "get" -> Script.Cmd.Action.Get(name, str) @@ -49,7 +54,7 @@ open class SExprToAst { return when(exp.vals.first().symbolStr()) { "assert_return" -> Script.Cmd.Assertion.Return(toAction(mult), - exp.vals.drop(2).map { toExprMaybe(it as SExpr.Multi, ExprContext(emptyMap())) }) + exp.vals.drop(2).map { toExprMaybe(it as SExpr.Multi, ExprContext.empty) }) "assert_return_canonical_nan" -> Script.Cmd.Assertion.ReturnNan(toAction(mult), canonical = true) "assert_return_arithmetic_nan" -> @@ -176,7 +181,7 @@ open class SExprToAst { var innerCtx = ctx.copy(blockDepth = ctx.blockDepth + 1) exp.maybeName(opOffset)?.also { opOffset++ - innerCtx = innerCtx.copy(nameMap = innerCtx.nameMap + ("block:$it" to innerCtx.blockDepth)) + innerCtx = innerCtx.copy(nameMap = innerCtx.nameMap.add("block", it, innerCtx.blockDepth)) } val sigs = toBlockSigMaybe(exp, opOffset) @@ -233,7 +238,7 @@ open class SExprToAst { var (nameMap, exprsUsed, sig) = toFuncSig(exp, currentIndex, origNameMap, types) currentIndex += exprsUsed val locals = exp.repeated("local", currentIndex, { toLocals(it) }).mapIndexed { index, (nameMaybe, vals) -> - nameMaybe?.also { require(vals.size == 1); nameMap += "local:$it" to (index + sig.params.size) } + nameMaybe?.also { require(vals.size == 1); nameMap = nameMap.add("local", it, index + sig.params.size) } vals } currentIndex += locals.size @@ -250,7 +255,8 @@ open class SExprToAst { name = name, func = Node.Func(sig, locals.flatten(), instrs), importOrExport = maybeImpExp, - additionalFuncTypesToAdd = ctx.callIndirectNeverBeforeSeenFuncTypes + additionalFuncTypesToAdd = ctx.callIndirectNeverBeforeSeenFuncTypes, + nameMap = nameMap ) } @@ -268,7 +274,7 @@ open class SExprToAst { } else null to offset var nameMap = origNameMap val params = exp.repeated("param", offset, { toParams(it) }).mapIndexed { index, (nameMaybe, vals) -> - nameMaybe?.also { require(vals.size == 1); nameMap += "local:$it" to index } + nameMaybe?.also { require(vals.size == 1); nameMap = nameMap.add("local", it, index) } vals } val resultExps = exp.repeated("result", offset + params.size, this::toResult) @@ -395,7 +401,7 @@ open class SExprToAst { val maybeName = exp.maybeName(offset + opOffset) if (maybeName != null) { opOffset++ - innerCtx = innerCtx.copy(nameMap = innerCtx.nameMap + ("block:$maybeName" to innerCtx.blockDepth)) + innerCtx = innerCtx.copy(nameMap = innerCtx.nameMap.add("block", maybeName, innerCtx.blockDepth)) } val sigs = toBlockSigMaybe(exp, offset + opOffset) opOffset += sigs.size @@ -428,7 +434,7 @@ open class SExprToAst { opOffset++ exp.maybeName(offset + opOffset)?.also { opOffset++ - innerCtx = innerCtx.copy(nameMap = innerCtx.nameMap + ("block:$it" to ctx.blockDepth)) + innerCtx = innerCtx.copy(nameMap = innerCtx.nameMap.add("block", it, ctx.blockDepth)) } toInstrs(exp, offset + opOffset, innerCtx, false).also { ret += it.first @@ -522,7 +528,7 @@ open class SExprToAst { val exps = exp.vals.mapNotNull { it as? SExpr.Multi } // Eagerly build the names (for forward decls) - val (nameMap, eagerTypes) = toModuleForwardNameMapAndTypes(exps) + var (nameMap, eagerTypes) = toModuleForwardNameMapAndTypes(exps) mod = mod.copy(types = eagerTypes) fun Node.Module.addTypeIfNotPresent(type: Node.Type.Func): Pair { @@ -555,6 +561,7 @@ open class SExprToAst { Node.Import.Kind.Memory(kind) to (memoryCount++ to Node.ExternalKind.MEMORY) else -> throw Exception("Unrecognized import kind: $kind") } + mod = mod.copy( imports = mod.imports + Node.Import(module, field, importKind), exports = mod.exports + exportFields.map { @@ -579,11 +586,14 @@ open class SExprToAst { "elem" -> mod = mod.copy(elems = mod.elems + toElem(exp, nameMap)) "data" -> mod = mod.copy(data = mod.data + toData(exp, nameMap)) "start" -> mod = mod.copy(startFuncIndex = toStart(exp, nameMap)) - "func" -> toFunc(exp, nameMap, mod.types).also { (_, fn, impExp, additionalFuncTypes) -> + "func" -> toFunc(exp, nameMap, mod.types).also { (_, fn, impExp, additionalFuncTypes, localNameMap) -> if (impExp is ImportOrExport.Import) { handleImport(impExp.module, impExp.name, fn.type, impExp.exportFields) } else { if (impExp is ImportOrExport.Export) addExport(impExp, Node.ExternalKind.FUNCTION, funcCount) + if (includeNames) nameMap = nameMap.copy( + localNames = nameMap.localNames!! + (funcCount to localNameMap.getAllNamesByIndex("local")) + ) funcCount++ mod = mod.copy(funcs = mod.funcs + fn).addTypeIfNotPresent(fn.type).first mod = additionalFuncTypes.fold(mod) { mod, typ -> mod.addTypeIfNotPresent(typ).first } @@ -644,6 +654,15 @@ open class SExprToAst { if (mod.tables.size + mod.imports.count { it.kind is Node.Import.Kind.Table } > 1) throw IoErr.MultipleTables() + // Set the name map pieces if we're including them + if (includeNames) mod = mod.copy( + names = Node.NameSection( + moduleName = name, + funcNames = nameMap.funcNames!!, + localNames = nameMap.localNames!! + ) + ) + return name to mod } @@ -680,10 +699,14 @@ open class SExprToAst { var globalCount = 0 var tableCount = 0 var memoryCount = 0 - var namesToIndices = emptyMap() + var nameMap = NameMap( + names = emptyMap(), + funcNames = if (includeNames) emptyMap() else null, + localNames = if (includeNames) emptyMap() else null + ) var types = emptyList() fun maybeAddName(name: String?, index: Int, type: String) { - name?.let { namesToIndices += "$type:$it" to index } + name?.also { nameMap = nameMap.add(type, it, index) } } // All imports first @@ -711,12 +734,12 @@ open class SExprToAst { "memory" -> maybeAddName(kindName, memoryCount++, "memory") // We go ahead and do the full type def build here eagerly "type" -> maybeAddName(kindName, types.size, "type").also { _ -> - toTypeDef(it, namesToIndices).also { (_, type) -> types += type } + toTypeDef(it, nameMap).also { (_, type) -> types += type } } else -> {} } } - return namesToIndices to types + return nameMap to types } fun toOpMaybe(exp: SExpr.Multi, offset: Int, ctx: ExprContext): Pair? { @@ -753,7 +776,8 @@ open class SExprToAst { // First lookup the func sig val (updatedNameMap, expsUsed, funcType) = toFuncSig(exp, offset + 1, ctx.nameMap, ctx.types) // Make sure there are no changes to the name map - if (ctx.nameMap.size != updatedNameMap.size) throw IoErr.IndirectCallSetParamNames() + if (ctx.nameMap.size != updatedNameMap.size) + throw IoErr.IndirectCallSetParamNames() // Obtain the func index from the types table, the indirects table, or just add it var funcTypeIndex = ctx.types.indexOf(funcType) // If it's not in the type list, check the call indirect list @@ -910,7 +934,7 @@ open class SExprToAst { fun toVarMaybe(exp: SExpr, nameMap: NameMap, nameType: String): Int? { return exp.symbolStr()?.let { it -> if (it.startsWith("$")) - nameMap["$nameType:$it"] ?: + nameMap.get(nameType, it.drop(1)) ?: throw Exception("Unable to find index for name $it of type $nameType in $nameMap") else if (it.startsWith("0x")) it.substring(2).toIntOrNull(16) else it.toIntOrNull() @@ -1005,7 +1029,7 @@ open class SExprToAst { private fun SExpr.Multi.maybeName(index: Int): String? { if (this.vals.size > index && this.vals[index] is SExpr.Symbol) { val sym = this.vals[index] as SExpr.Symbol - if (!sym.quoted && sym.contents[0] == '$') return sym.contents + if (!sym.quoted && sym.contents[0] == '$') return sym.contents.drop(1) } return null } @@ -1028,5 +1052,26 @@ open class SExprToAst { return this.vals.first().requireSymbol(contents, quotedCheck) } + data class NameMap( + // Key prefixed with type then colon before actual name + val names: Map, + // Null if not including names + val funcNames: Map?, + val localNames: Map>? + ) { + val size get() = names.size + + fun add(type: String, name: String, index: Int) = copy( + names = names + ("$type:$name" to index), + funcNames = funcNames?.let { if (type == "func") it + (index to name) else it } + ) + + fun get(type: String, name: String) = names["$type:$name"] + + fun getAllNamesByIndex(type: String) = names.mapNotNull { (k, v) -> + k.takeIf { k.startsWith("$type:") }?.let { v to k.substring(type.length + 1) } + }.toMap() + } + companion object : SExprToAst() } diff --git a/compiler/src/main/kotlin/asmble/io/StrToSExpr.kt b/compiler/src/main/kotlin/asmble/io/StrToSExpr.kt index a9b1f7e..d0d14c4 100644 --- a/compiler/src/main/kotlin/asmble/io/StrToSExpr.kt +++ b/compiler/src/main/kotlin/asmble/io/StrToSExpr.kt @@ -18,6 +18,13 @@ open class StrToSExpr { data class Error(val pos: Pos, val msg: String) : ParseResult() } + fun parseSingleMulti(str: CharSequence) = parse(str).let { + when (it) { + is ParseResult.Success -> (it.vals.singleOrNull() as? SExpr.Multi) ?: error("Not a single multi-expr") + is ParseResult.Error -> error("Failed parsing at ${it.pos.line}:${it.pos.char} - ${it.msg}") + } + } + fun parse(str: CharSequence): ParseResult { val state = ParseState(str) val ret = mutableListOf() diff --git a/compiler/src/main/kotlin/asmble/run/jvm/Module.kt b/compiler/src/main/kotlin/asmble/run/jvm/Module.kt index 5dcf92b..8247eea 100644 --- a/compiler/src/main/kotlin/asmble/run/jvm/Module.kt +++ b/compiler/src/main/kotlin/asmble/run/jvm/Module.kt @@ -116,9 +116,9 @@ interface Module { } // Global imports - val globalImports = mod.imports.mapNotNull { - if (it.kind is Node.Import.Kind.Global) ctx.resolveImportGlobal(it, it.kind.type) - else null + val globalImports = mod.imports.flatMap { + if (it.kind is Node.Import.Kind.Global) ctx.resolveImportGlobals(it, it.kind.type) + else emptyList() } constructorParams += globalImports diff --git a/compiler/src/main/kotlin/asmble/run/jvm/RunErr.kt b/compiler/src/main/kotlin/asmble/run/jvm/RunErr.kt index 47619e0..aa81251 100644 --- a/compiler/src/main/kotlin/asmble/run/jvm/RunErr.kt +++ b/compiler/src/main/kotlin/asmble/run/jvm/RunErr.kt @@ -55,4 +55,12 @@ sealed class RunErr(message: String, cause: Throwable? = null) : RuntimeExceptio override val asmErrString get() = "unknown import" override val asmErrStrings get() = listOf(asmErrString, "incompatible import type") } + + class ImportGlobalInvalidMutability( + val module: String, + val field: String, + val expected: Boolean + ) : RunErr("Expected imported global $module::$field to have mutability as ${!expected}") { + override val asmErrString get() = "incompatible import type" + } } \ No newline at end of file diff --git a/compiler/src/main/kotlin/asmble/run/jvm/ScriptContext.kt b/compiler/src/main/kotlin/asmble/run/jvm/ScriptContext.kt index f1faf3f..f602d72 100644 --- a/compiler/src/main/kotlin/asmble/run/jvm/ScriptContext.kt +++ b/compiler/src/main/kotlin/asmble/run/jvm/ScriptContext.kt @@ -277,10 +277,12 @@ data class ScriptContext( return Module.Compiled(mod, classLoader.fromBuiltContext(ctx), name, ctx.mem) } - fun bindImport(import: Node.Import, getter: Boolean, methodType: MethodType): MethodHandle { + fun bindImport(import: Node.Import, getter: Boolean, methodType: MethodType) = bindImport( + import, if (getter) "get" + import.field.javaIdent.capitalize() else import.field.javaIdent, methodType) + + fun bindImport(import: Node.Import, javaName: String, methodType: MethodType): MethodHandle { // Find a method that matches our expectations val module = registrations[import.module] ?: throw RunErr.ImportNotFound(import.module, import.field) - val javaName = if (getter) "get" + import.field.javaIdent.capitalize() else import.field.javaIdent val kind = when (import.kind) { is Node.Import.Kind.Func -> WasmExternalKind.FUNCTION is Node.Import.Kind.Table -> WasmExternalKind.TABLE @@ -295,8 +297,18 @@ data class ScriptContext( bindImport(import, false, MethodType.methodType(funcType.ret?.jclass ?: Void.TYPE, funcType.params.map { it.jclass })) - fun resolveImportGlobal(import: Node.Import, globalType: Node.Type.Global) = - bindImport(import, true, MethodType.methodType(globalType.contentType.jclass)) + fun resolveImportGlobals(import: Node.Import, globalType: Node.Type.Global): List { + val getter = bindImport(import, true, MethodType.methodType(globalType.contentType.jclass)) + // Whether the setter is present or not defines whether it is mutable + val setter = try { + bindImport(import, "set" + import.field.javaIdent.capitalize(), + MethodType.methodType(Void.TYPE, globalType.contentType.jclass)) + } catch (e: RunErr.ImportNotFound) { null } + // Mutability must match + if (globalType.mutable == (setter == null)) + throw RunErr.ImportGlobalInvalidMutability(import.module, import.field, globalType.mutable) + return if (setter == null) listOf(getter) else listOf(getter, setter) + } fun resolveImportMemory(import: Node.Import, memoryType: Node.Type.Memory, mem: Mem) = bindImport(import, true, MethodType.methodType(Class.forName(mem.memType.asm.className))). diff --git a/compiler/src/test/kotlin/asmble/SpecTestUnit.kt b/compiler/src/test/kotlin/asmble/SpecTestUnit.kt index 4c8f55e..df24896 100644 --- a/compiler/src/test/kotlin/asmble/SpecTestUnit.kt +++ b/compiler/src/test/kotlin/asmble/SpecTestUnit.kt @@ -13,8 +13,8 @@ class SpecTestUnit(name: String, wast: String, expectedOutput: String?) : BaseTe override val shouldFail get() = name.endsWith(".fail") override val defaultMaxMemPages get() = when (name) { - "nop"-> 20 - "resizing" -> 830 + "nop" -> 20 + "memory_grow" -> 830 "imports" -> 5 else -> 1 } diff --git a/compiler/src/test/kotlin/asmble/compile/jvm/LargeDataTest.kt b/compiler/src/test/kotlin/asmble/compile/jvm/LargeDataTest.kt new file mode 100644 index 0000000..b938d39 --- /dev/null +++ b/compiler/src/test/kotlin/asmble/compile/jvm/LargeDataTest.kt @@ -0,0 +1,45 @@ +package asmble.compile.jvm + +import asmble.TestBase +import asmble.ast.Node +import asmble.run.jvm.ScriptContext +import asmble.util.get +import org.junit.Test +import java.nio.ByteBuffer +import java.util.* +import kotlin.test.assertEquals + +class LargeDataTest : TestBase() { + @Test + fun testLargeData() { + // This previously failed because string constants can't be longer than 65536 chars. + // We create a byte array across the whole gambit of bytes to test UTF8 encoding. + val bytesExpected = ByteArray(70000) { ((it % 255) - Byte.MIN_VALUE).toByte() } + val mod = Node.Module( + memories = listOf(Node.Type.Memory( + limits = Node.ResizableLimits(initial = 2, maximum = 2) + )), + data = listOf(Node.Data( + index = 0, + offset = listOf(Node.Instr.I32Const(0)), + data = bytesExpected + )) + ) + val ctx = ClsContext( + packageName = "test", + className = "Temp" + UUID.randomUUID().toString().replace("-", ""), + mod = mod, + logger = logger + ) + AstToAsm.fromModule(ctx) + val cls = ScriptContext.SimpleClassLoader(javaClass.classLoader, logger).fromBuiltContext(ctx) + // Instantiate it, get the memory out, and check it + val field = cls.getDeclaredField("memory").apply { isAccessible = true } + val buf = field[cls.newInstance()] as ByteBuffer + // Grab all + 1 and check values + val bytesActual = ByteArray(70001).also { buf.get(0, it) } + bytesActual.forEachIndexed { index, byte -> + assertEquals(if (index == 70000) 0.toByte() else bytesExpected[index], byte) + } + } +} \ No newline at end of file diff --git a/compiler/src/test/kotlin/asmble/compile/jvm/NamesTest.kt b/compiler/src/test/kotlin/asmble/compile/jvm/NamesTest.kt new file mode 100644 index 0000000..51f327e --- /dev/null +++ b/compiler/src/test/kotlin/asmble/compile/jvm/NamesTest.kt @@ -0,0 +1,38 @@ +package asmble.compile.jvm + +import asmble.TestBase +import asmble.io.SExprToAst +import asmble.io.StrToSExpr +import asmble.run.jvm.ScriptContext +import org.junit.Test +import java.util.* + +class NamesTest : TestBase() { + @Test + fun testNames() { + // Compile and make sure the names are set right + val (_, mod) = SExprToAst.toModule(StrToSExpr.parseSingleMulti(""" + (module ${'$'}mod_name + (import "foo" "bar" (func ${'$'}import_func (param i32))) + (type ${'$'}some_sig (func (param ${'$'}type_param i32))) + (func ${'$'}some_func + (type ${'$'}some_sig) + (param ${'$'}func_param i32) + (local ${'$'}func_local0 i32) + (local ${'$'}func_local1 f64) + ) + ) + """.trimIndent())) + val ctx = ClsContext( + packageName = "test", + className = "Temp" + UUID.randomUUID().toString().replace("-", ""), + mod = mod, + logger = logger + ) + AstToAsm.fromModule(ctx) + val cls = ScriptContext.SimpleClassLoader(javaClass.classLoader, logger).fromBuiltContext(ctx) + // Make sure the import field and the func are present named + cls.getDeclaredField("import_func") + cls.getDeclaredMethod("some_func", Integer.TYPE) + } +} \ No newline at end of file diff --git a/compiler/src/test/kotlin/asmble/io/NamesTest.kt b/compiler/src/test/kotlin/asmble/io/NamesTest.kt new file mode 100644 index 0000000..3937073 --- /dev/null +++ b/compiler/src/test/kotlin/asmble/io/NamesTest.kt @@ -0,0 +1,47 @@ +package asmble.io + +import asmble.ast.Node +import org.junit.Test +import kotlin.test.assertEquals + +class NamesTest { + @Test + fun testNames() { + // First, make sure it can parse from sexpr + val (_, mod1) = SExprToAst.toModule(StrToSExpr.parseSingleMulti(""" + (module ${'$'}mod_name + (import "foo" "bar" (func ${'$'}import_func (param i32))) + (type ${'$'}some_sig (func (param ${'$'}type_param i32))) + (func ${'$'}some_func + (type ${'$'}some_sig) + (param ${'$'}func_param i32) + (local ${'$'}func_local0 i32) + (local ${'$'}func_local1 f64) + ) + ) + """.trimIndent())) + val expected = Node.NameSection( + moduleName = "mod_name", + funcNames = mapOf( + 0 to "import_func", + 1 to "some_func" + ), + localNames = mapOf( + 1 to mapOf( + 0 to "func_param", + 1 to "func_local0", + 2 to "func_local1" + ) + ) + ) + assertEquals(expected, mod1.names) + // Now back to binary and then back and make sure it's still there + val bytes = AstToBinary.fromModule(mod1) + val mod2 = BinaryToAst.toModule(bytes) + assertEquals(expected, mod2.names) + // Now back to sexpr and then back to make sure the sexpr writer works + val sexpr = AstToSExpr.fromModule(mod2) + val (_, mod3) = SExprToAst.toModule(sexpr) + assertEquals(expected, mod3.names) + } +} \ No newline at end of file