Release Notes for ModeShape 4.0.0.Beta1

This document outlines the changes that were made in ModeShape 4.0.0.Beta1. We hope you enjoy it!

What's new

4.0.0.Beta1 provides a fast, elastic, distributed hierarchical database that clients work with via the standard JCR 2.0 (JSR-283) API. ModeShape 4 is a major upgrade over 3.x and offers significant improvements in clustering, performance, query, and events. All JCR 2.0 features are supported, and ModeShape 4 has complete integration with Wildfly 8, allowing deployed applications to simply lookup and use repositories managed by ModeShape's service.

As of 4.0.0.Alpha1, ModeShape is licensed under the Apache Software License, 2.0.

This is the first beta release of the 4.0 stream, and it includes 46 bug fixes and several new features. There is now support for explicit single-column indexes stored locally on the file system. As a result, the index provider SPI first released in an earlier alpha has changed slightly, though we expect it to remain unchanged from this point forward. We've also extended the JCR-SQL2 query language with a new pseudocolumn, "mode:id", that provides access to exactly the same value as "Node.getIndentifier()" would via the API. Like all pseudocolumns, it can be used in WHERE constraints and JOIN criteria.

The first alpha release introduced a new query engine that allows clients to explicitly define the indexes used in the query system, and the second alpha release brought minor changes to the Service Provider Interface (SPI) for query providers and introduced a programmatic API and configuration modifications for defining indexes, although no complete query providers are included (see next release). Alpha2 had support for the JCR event journal feature, allowing applications to poll for changes that occurred during time ranges. This is a useful alternative to listeners that may be expensive or time-consuming. Alpha3 introduced a new event system and our new ring buffer that is substantially faster than what we had in 3.x; of course, there's no change in the event APIs so your listener implementations will continue to work unchanged. Alpha4 fixed a number of issues and introduced the newly redesigned Repository Explorer web application that can be deployed in a web server alongside ModeShape, including on Wildfly 8.

What to test

Since this is the first beta release, all features targeted to 4.0 are complete and are suitable for testing. It is not a stable release, so please do not put 4.0.0.Beta1 into production.

We would like to get as much feedback as possible, so we do ask that our community do testing with 4.0.0.Beta1 to help us identify problems. Specifically, we ask that you test the following areas:

• JDK - ModeShape now requires JDK 7. We've not yet begun testing with Java 8, but we'd be happy to hear about it if you do.
• Queries - the new query engine passes all of our regression tests. Without explicit indexes, all queries are expected to work properly but may be slow (except for very tiny repositories). You can explicitly define indexes via the configuration files or programmatically via ModeShape's public API. In fact, it is recommended that you define indexes that can be used in each of your queries, and doing so will make those queries much faster. The query plan contains information about all indexes considered by the query engine as well the index (if any) selected for use in each part of the query; please use the query plan to understand whether your indexes are being used.
• Clustering - ModeShape no longer has a clustering section in its configuration, since we simply piggyback on top of Infinispan's clustering setup. We've also upgraded to a newer version of JGroups.
• Journalling - Try enabling journaling and verify it works and does not affect performance. Then try using the JCR Event Journal feature.
• Infinispan - We've moved to Infinispan 6.0.1.Final, which is faster and has new cache stores. Some older and poorly-performaing cache stores are no longer valid, so check out the new file-based cache stores. Also, the LevelDB cache store is supposedly very fast.
• Backup and restore - given that some older Infinispan cache stores are no longer supported, in order to test migrating 3.x repositories to 4.0 you will need to use ModeShape's backup and restore feature. If you don't regularly use that, please test it with your repository. Just be sure not to overwrite any 3.x repositories.
• Bugs - we've fixed a number of bugs reported against and fixed in 3.x; see the list below for details. All these are ready for testing.
• Configuration - comments are now allowed in our JSON configuration.

Features

ModeShape 4.0.0.Beta1 has these features:

• ModeShape uses Infinispan for all caching and storage, giving a powerful and flexible foundation for creating JCR repositories that are fast, scalable, and highly available. Infinispan offers a great deal of storage options (via cache loaders), but using Infinispan as a distributed, mulit-site, in-memory data grid provides incredible scalability and performance.
• Strongly consistent. ModeShape is atomic, consistent, isolated and durable (ACID), so writing applications is very natural. Applications can even use JTA transactions.
• Fast. ModeShape 4 is just plain seriously fast, and performance is all-around faster than earlier version.
• Larger content. ModeShape 4 can store and access the content so that a node can have hundreds of thousands (or more!) of child nodes (even with same-name-siblings) yet still be incredibly fast. Additionally, repositories can scale to many millions of nodes and be deployed across many processes.
• Simplified configuration. There is no more global configuration of the engine; instead, each repository is configured with a separate JSON file, which must conform to a ModeShape-specific JSON Schema and can be validated by ModeShape prior to use. Repository configurations can even be changed while the repository is running (some restrictions apply), making it possible to add/change/remove sequencers, authorization providers, and many other configuration options while the repository is in use.
• Elastic. Add processes to scale out, without having to have a single coordinator.
• Deploy, start, stop and undeploy repositories while the engine is running and while and other repositories are still in use.
• Sessions immediately see all changes persisted/committed by other sessions, although transient changes made by the session always take precedence.
• Monitoring API with over a dozen metrics.
• Sequencing SPI that uses the JCR API to get at the content being processed and create/update the derived content. Sequencers can also dynamically register namespaces and node types. Now it's easy to create custom sequencers.
• Connector SPI that defines how external systems are accessed and optionally updated to project the external information into the repository as regular nodes.
• Simple API for implementing custom MIME type detectors. ModeShape still has a built-in Tika-based detector that determines MIME types using the filename extensions and binary content.
• Simple API for implementing custom text extractors, which extracts from binary values searchable text used in full-text searches and queries.
• Ability to store binary values of any sizes, with a separate facility for storing these on the file system, in Infinispan caches, in relational DBMSes (via JDBC), and in MongoDB. Custom stores are also possible.
• Public API interfaces and methods that were deprecated in 2.7.0.Final (or later) have been removed. There weren't many of these; most of the ModeShape API remains the same as 2.x.
• Integration with JBoss AS 7. ModeShape runs as an integrated subsystem within AS7, and the AS7 tooling can be used to define and manage repositories independently of each other and while the server is running.
• Local and remote JDBC drivers for issuing JCR-SQL2 queries and getting database metadata via the JDBC API
• Use the RESTful API to talk to ModeShape repositories from non-Java and non-local applications
• Use the CMIS API to talk to ModeShape repositories
• Use WebDAV to mount ModeShape repositories as file servers
• Visual repository explorer web application

All of the JCR 2.0 features previously supported in 2.x are currently supported:

Accessing the Repository

• RepositoryFactory access
• JNDI registration of Repository
• JAAS Authentication
• Servlet Authentication
• Custom Authentication

Namespaces

• Session Remapping
• Permanent Addition/Deletion

Reading Repository Content

• Traversal Access
• Direct Access
• Same-Name Siblings
• Multi-Value Properties
• All Property Types Supported
• Property Type Conversion

Writing Repository Content

• Create/Update/Delete Nodes
• Create/Update/Delete Properties (Through Parent Nodes)
• Moving, Copying, Cloning
• Adding/Removing Mixins
• Referential integrity enforcement

Query / Search

• XPath
• JCR-SQL
• JCR-SQL2
• JCR-QOM
• Full-Text Search

Importing/Exporting Repository Content

• System View Import/Export
• Document View Import/Export

Node Types

• Inheritance Among Node Types
• Discovering available Node Types
• Discovering the Node Types of a Node
• Discovering the Definition of a Node Type
• Property Constraints
• Automatic Item Creation
• Predefined standard Node Types
• Custom Node Type Registration (CND-Based and and JCR 2.0 API Template-Based)

Repository Metadata under System Node

• Permanent Namespace Mappings
• Node Types (Built-In and User-Registered)
• Active Locks

Other JCR Optional Features

• Observation
• Locking
• Versioning
• Shareable nodes
• Access controls

Content Storage Options

• In-memory (local, replicated, and distributed)
• Relational databases (via JDBC), including in-memory, file-based, or remote
• LevelDB
• File system
• Cassandra
• Cloud storage (e.g., Amazon's S3, Rackspace's Cloudfiles, or any other provider supported by JClouds)
• Remote Infinispan

Binary Storage Options

• File system
• JDBC database
• Infinispan
• MongoDB
• Chained binary stores

ModeShape also has features that go beyond the JCR API:

ModeShape Federation Connectors

• File system connector (read and write)
• Git repository connector (read-only)
• CMIS reposiotry connector (read and write, tech preview)
• JDBC metadata connector (read-only)

ModeShape Sequencers

• Compact Node Definition (CND) Sequencer
• DDL Sequencer
• Image Sequencer
• Java Source Sequencer
• Java Class Sequencer
• MP3 Sequencer
• MS Office Sequencer
• Text Sequencers (Delimited and Fixed Width)
• XML Sequencer
• XML Schema Document (XSD) Sequencer
• Web Service Definition Lanaguage (WSDL) 1.1 Sequencer
• Zip File Sequencer (also WARs, JARs, and EARs)
• Teiid Relational Model Sequencer
• Teiid VDB Sequencer

ModeShape Deployment/Access Models

• JNDI-Based Deployment
• Deploy as a subsystem in JBoss AS7, with access to repositories via @Resource injection
• Deploy to other containers using ModeShape's JCA adapter
• Access through two RESTful Services (the 2.x-compatible API and a new improved API)
• Access through WebDAV Service
• Access through CMIS Service
• Access through visual repository explorer web application
• Local and remote JDBC drivers for accessing ModeShape content through JDBC API and JCR-SQL2 queries
• Embedded (in Server or JEE Archive) Deployment
• JTA support, allowing Sessions to participate in XA and container-managed transactions
• OSGi-Compatible Archives

Other ModeShape features

• Repository-wide backup and restoration
• Explicitly-defined indexes
• Automatic MIME type detection of binary content
• Asynchronous sequencing operations, within completion notified via events

Bug Fixes, Features, and other Issues

The following are the bugs, features and other issues that have been fixed in the 4.0.0.Beta1 release:

Bug

• [MODE-2054] - Invalid query results for sql with outer join and OR join criteria
• [MODE-2158] - web-explorer is leaking connections
• [MODE-2174] - JCR Query in Modeshape returns no result when empty string used in comparison
• [MODE-2178] - Limitations and problems in the new query engine
• [MODE-2209] - New query engine does not properly enforce permissions
• [MODE-2224] - Java Source File Sequencer Problems When Sequencing Interfaces, Typed Collections, And Nested Types
• [MODE-2238] - Repository Explorer's query results only show 1 line at a time
• [MODE-2243] - Accessing WebDAV as network drive via the Windows client does not work
• [MODE-2245] - Repository Explorer must be secured
• [MODE-2247] - Intersect and Except set queries are not implemented
• [MODE-2248] - Repository Explorer doesn't work in Wildfly
• [MODE-2249] - Repository Explorer doesn't work in Wildfly
• [MODE-2250] - Only one SNS node is shown when using items REST API with depth=-1
• [MODE-2252] - Creating 22 files in a directory in a filesystem projection fails
• [MODE-2253] - Backup/Restore does not work in JBoss EAP
• [MODE-2254] - DefaultValueFactory.createDate() throws StringIndexOutOfBoundsException for 20 or 21 character strings
• [MODE-2255] - Files in queryable federated filesystem not found by queries
• [MODE-2256] - Moving/Renaming an item on a federated filesystem results in properties being lost
• [MODE-2258] - Incorrect encoding of non-ASCII characters in REST API URLs
• [MODE-2259] - The ModeShape subsystem cannot be added via CLI to multiple profiles in domain mode
• [MODE-2260] - Broken navigation on Path
• [MODE-2261] - Can't execute queries with multiple selectors via REST API
• [MODE-2263] - Trying to import metadata using TeiidDesigner, and getting permission denied
• [MODE-2264] - InterruptedException Sequencing Multiple Files In Test Using Java Sequencer
• [MODE-2267] - Query constraints on node name and local name are swapped
• [MODE-2268] - The JcrObservationManager doesn't filter local/nonLocal events correctly
• [MODE-2269] - javax.security.jacc dependency should have the scope 'provided'
• [MODE-2271] - Improve performance of getting path of node that is below a large ancestor
• [MODE-2272] - Java Sequencer Should Be Thread-Safe
• [MODE-2273] - Navigation using FORWARD/BACKWARD browser buttons
• [MODE-2274] - Java sequencer doesn't handle files with no package statement
• [MODE-2275] - javax.jcr.query.InvalidQueryException: Table does not exist. when adding multiple nodetypes
• [MODE-2278] - Registering unsupported namespaces should result in NamespaceException
• [MODE-2279] - Index "kind" configuration attribute is incorrectly parsed
• [MODE-2281] - Assigning nodeType nt:version (and probably some others from jcr:system) breaks things
• [MODE-2283] - Losing backreferences after modify property reference value

Enhancement

• [MODE-2157] - The configuration option to disable queries is not needed and should be removed
• [MODE-2232] - Repository Explorer links should be standard colors but should not be underlined
• [MODE-2233] - Repository Explorer's administrative functions should be separated onto administrative page
• [MODE-2234] - Repository Explorer's save button should be moved closer to where changes are actually made
• [MODE-2265] - Evaluate whether to replace transient Infinispan as the workspace cache
• [MODE-2266] - Create simple scalability and performance tests
• [MODE-2276] - Improve performance of adding node under parent
• [MODE-2277] - Improve performance of isCheckedOut

Feature Request

• [MODE-1671] - Provide a way to specify a node's identifier in queries
• [MODE-2160] - Store indexes on the local file system (using MapDB)

Task

• [MODE-2135] - Event journal should use MapDB's append-only file
• [MODE-2185] - Remove old REST service

The following are the bugs, features and other issues that have been fixed in the 4.0.0.Alpha4 release:

Bug

• MODE-2229 - OutOfMemory error when running on 32bit OS
• MODE-2228 - ModeShape subsystem for Wildfly (EAP) should default to "readonly" for anonymous roles
• MODE-2221 - ModeShape 3.8.0 not releasing JDBC connections to the pool
• MODE-2220 - using uppercase() on transformwithSQL.transformselectSQL generates error
• MODE-2246 - Queries with CONTAINS clause fail
• MODE-2226 - ModeShape's fork channel prevents multiple clustered repositories from starting
• MODE-2240 - Getting binary store errors on Windows (s-ramp integration tests)
• MODE-2196 - Windows Explorer WebDAV Client Requests "desktop.ini" as Workspace
• MODE-2214 - ModeShape-Schematic JSON Reader hangs on incorrect json
• MODE-2216 - Move operation on cluster causes InvalidItemStateException exception
• MODE-2217 - Child nodes order is not respected when importing initial content
• MODE-2236 - Repository Explorer's "Node Types" button results in error (?) popup, but it works
• MODE-2218 - Importing metadata via JDBC driver doesn't return all columns
• MODE-2219 - errors when publishing teiid models

Component Upgrade

• MODE-2239 - Upgrade to MapDB 1.0.3

Enhancement

• MODE-2242 - Change ModeShape to use the ApplicationRealm
• MODE-2211 - Simplify ACL checks and make them more efficient
• MODE-2237 - Repository Explorer's query page should default to "JCR-SQL2" query language
• MODE-2230 - Breadcrumb path in RepositoryExplorer needs to clearly show as separate links for each segment

Feature Request

• MODE-2065 - Support installing ModeShape as a subsystem into Wildfly
• MODE-2225 - Custom AuthorizationProvider is not invoked for each child node when iterating through the child nodes of a parent node
• MODE-2208 - Add support for EAP 6.3 RBAC features
• MODE-2241 - Creating reference properties through initial content
• MODE-2093 - Web explorer should provide modern view on content

The following are the bugs, features and other issues that have been fixed in the 4.0.0.Alpha3 release:

Bug

• MODE-2200 - decrementBinaryReferenceCount() fails when InfinispanBinaryStore and Federation are enabled
• MODE-2203 - NPE in abstract node
• MODE-2202 - Changes made by sessions with user trasactions enabled are leaked to other sessions
• MODE-2201 - FileRepositoryLookup doesn't work with JAR protocol URLs
• MODE-2183 - The XSD Sequencer Creates Two Nodes For An Unqualified XSD Lexicon Elements
• MODE-2207 - Initialize on sequencers are only called when automatic sequencing is used
• MODE-2206 - Session.move fails even if existing destination node is removed in the same session
• MODE-2210 - Session.getNodeByIdentifier does not check ACLs
• MODE-2194 - Database binary stores should not hold on to PreparedStatements
• MODE-2173 - ACLs are not checked when returning nodes from queries
• MODE-2190 - Lock Garage Collection can cause NPE as locks are removed by sessions

Enhancement

• MODE-2088 - Event bus should internally have a ring buffer shared used by all listeners

Feature Request

• MODE-2212 - Restarting a repository with generated namespace prefixes fails
• MODE-2065 - Support installing ModeShape as a subsystem into Wildfly

Task

• MODE-2205 - Image Sequencer examples/tests are out of date.
• MODE-2198 - Fix clustering quickstart
• MODE-2197 - Update Apache POI library to 3.10-FINAL

The following are the bugs, features and other issues that have been fixed in the 4.0.0.Alpha2 release:

Bug

• MODE-2017 - Move doesn't work with CMIS connector
• MODE-2125 - Exceptions when Teiid's JDBC translator uses ModeShape's JDBC DatabaseMetaData implementation
• MODE-2193 - Calling Node.getNodes() throws java.util.NoSuchElementException when applying ACLs
• MODE-2180 - Same named properties allowed in REST API
• MODE-2179 - Error using MS SQL Server 2008 DB as binary store
• MODE-2186 - Iterating children via the getNodes() methods does not check ACLs
• MODE-2189 - File system connector's generated events should match similar operations with internal nodes
• MODE-2176 - FileSystemConnector config not updated when repository.json is updated
• MODE-2171 - Importing document view XML with ACLs enabled raises exception
• MODE-2192 - Error on importXML

Enhancement

• MODE-2182 - Inefficient creation of nodes with large binary content

Feature Request

• MODE-2019 - Support JCR event journal functionality
• MODE-2188 - Manage custom indexes

Task

• MODE-2191 - Rename the federation SPI package

The following are the bugs, features and other issues that have been fixed in the 4.0.0.Alpha1 release:

Bug

• MODE-1696 - Remove support for legacy (JSR-170) CND keywords not compliant with JSR-283
• MODE-2146 - The modeshape-jca module is incorrectly packaged
• MODE-2147 - External nodes should not allow mix:versionable
• MODE-2140 - Removing nodetype succeeds while it is in use with own query configuration
• MODE-2149 - Adding wrong constraint type causes data inconsistency
• MODE-2181 - The same name child ignored while creating node using REST API
• MODE-2187 - Unable to build with -Dassembly due to failure in JBoss AS kit
• MODE-2153 - ReferentialIntegrityException When Remove A Version
• MODE-2152 - NullPointerException When Removing A Version
• MODE-2150 - Create node type with boolean type and constraints causes checker exception
• MODE-2156 - Setting the same value for a reference property multiple times, increases the referrer count each time
• MODE-2154 - Text extractor ignores text if write limit is exceeded
• MODE-2175 - Incorrect repository descriptor "java.repository.vendor.url"
• MODE-2177 - Current JavaDoc stylesheet is not compatible with the javadoc tool from Java 7
• MODE-2170 - Problem with saving multi-value property using REST 3.x
• MODE-2172 - Importing document view XML that contains checked-in nodes produces assertion error
• MODE-2144 - Garbage Collection using DatabaseBinaryStore deletes all entries
• MODE-2136 - Embedded BOM breaks Apache Avro dependency
• MODE-2139 - Removing protected property succeeds
• MODE-2138 - JCR-SQL2 query throws NPE if ordering by [jcr:path]
• MODE-2067 - Invalid bundle packaging and export

Component Upgrade

• MODE-2063 - Upgrade to Lucene 4.x and Hibernate Search 5.x
• MODE-2066 - Upgrade to Infinispan 6.x

Enhancement

• MODE-1839 - Add posibility to remove version history when versionable node has only root version
• MODE-2167 - Removing all access control policies should disable the ACL checks
• MODE-2164 - ModeShape configuration variables should use environment variables, too
• MODE-2165 - Exception message should be clearer when ModeShape is looking for cache with wrong name
• MODE-2079 - Use JGroups fork channel for all events
• MODE-2184 - Add public API methods to close query results
• MODE-2081 - Change license to ASL 2.0
• MODE-2080 - Use safer global lock during repository initialization
• MODE-2163 - Add support in JCR-SQL2 for ordering with nulls first or last
• MODE-1535 - ModeShape should configure Hibernate Search's NRTIndexManager for faster (non-clustered) performance
• MODE-2091 - Improve DDL parser error resolution

Feature Request

• MODE-2040 - Generate Node Events for Federated Files
• MODE-1698 - JCR 2.0 strict mode
• MODE-2065 - Support installing ModeShape as a subsystem into Wildfly
• MODE-2169 - Package java docs into a zip
• MODE-2041 - Require JDK 7 for compiling/building ModeShape 4.0
• MODE-2082 - Support comments in repository configuration JSON files
• MODE-1869 - Embed the Teiid relational query engine
• MODE-1683 - Add event journal
• MODE-2018 - Query engine should work with no indexes and with defined indexes

Quality Risk

• MODE-2142 - Create namespace with existing URI causing server startup to fail
• MODE-2141 - Unregistering namespace succeeds while nodetypes are in use

Task

• MODE-2145 - Upgrade JUnit to 4.11
• MODE-2023 - Refactor query and indexing functionality behind interfaces
• MODE-2148 - Use CheckStyle to verify proper formatting of source code
• MODE-2143 - Change build to use IPv6 by default
• MODE-2086 - Refactor the modeshape-schematic package
• MODE-1457 - Remove PropertyType.UUID type
• MODE-1604 - Convert commented-out tests in AS7 subsystem to use DMR rather than JSON
• MODE-2134 - Clean up BOMs to remove most third-party dependency specifications
• MODE-1372 - Internal RepositoryCache CreateNode events should include node type info for new node and parent