Perhaps So

How Speculoos addresses issues raised in 'Maybe Not'

Rich Hickey presented Maybe Not at the 2018 Clojure conj. If I correctly understand his presentation, he identified some flaws in clojure.spec.alpha, the Clojure distribution's built-in library for specifying and validating data. Mr Hickey highlighted three issues.

Representing partial information in an aggregate data structure.
Specifying partial information in an aggregate data structure.
Validating partial information in an aggregate data structure.

He was apparently not satisfied with the way spec.alpha handles these three issues.

The Speculoos library is an experiment to see if it is possible to perform the same tasks as spec.alpha using literal specifications. Due to some of the implementation details and policies I chose, the library has some emergent properties that end up neatly handling those three issues.

Efficiently using Speculoos requires remembering three mottos.

Validate scalars separately from validating collections.
Shape the specification to mimic the data.
Ignore un-paired predicates and un-paired datums.

If we follow those three mottos, handling partial information is straightforward. Briefly, Speculoos specifies and validates scalars separately from specifying and validating collections. A scalar specification describes the properties of the datums themselves. The presence or absence of a scalar is a completely separate concern and falls under the jurisdiction of specifying and validating the collection. By separating the two concerns, Speculoos seamlessly handles partial information while avoiding the issues that befall spec.alpha.

Related: clojure.spec.alpha side-by-side comparison to Speculoos.

Let's examine each issue in more detail.

Representing partial information

Representing partial data is not specifically the purview of the Speculoos library, but of Clojure itself. We'll discuss partial information only in order to supply us with examples for later.

Mr Hickey highlights the fact that idiomatic Clojure merely excludes missing information instead of 'holding a slot' with a nil. Imagine data about a person that could include their first name, last name, and and their age. Here's an example of 'complete' data.

{:first-name "Albert"
 :last-name "Einstein"
 :age 76}

The following example of partial data, with nil associated to 'missing' :age information, is atypical.

{:first-name "Isaac"
 :last-name "Newton"
 :age nil}

The more idiomatic way to represent partial information about a person involves merely leaving off the person's age.

{:first-name "Maria"
 :last-name "Göppert-Mayer"}

Specifying and validating partial information

A Speculoos specification is a plain Clojure data structure containing predicates. The specification's shape mimics the shape of the data (Motto #2). Professor Einstein's data is a map with keys :first-name, :last-name, and :age. The Speculoos specification for that data might look like this.

{:first-name string?
 :last-name string?
 :age int?}

The specification is likewise a map with those same keys, i.e., the same 'shape', with predicates string? and int? replacing datums. Speculoos assembles pairs of datums and predicates and reports if the datums satisfy their corresponding predicates and returns true/false.

All of Speculoos' validating functions have a similar signature. The data is the first argument, the specification is the second argument.

(valid-scalars? data
                specification)

I'll be printing the specification directly below the data to visually emphasize how the shape of the specification mimics the shape of the data (Motto #2).

(Speculoos offers a verbose variant if we need to see details of the validation.)

Validating complete data

(require '[speculoos.core :refer [valid-scalars?]])


(valid-scalars? {:first-name "Albert", :last-name "Einstein", :age 76}
                {:first-name string?, :last-name string?, :age int?})
;; => true

valid-scalars? systematically walks through both the data and specification, and where it finds a datum paired with a predicate, it validates.

"Albert" at :first-name in the data (upper row) satisfies string? at :first-name in the specification (lower row),
"Einstein" at :last-name in the data satisfies string? at :last-name in the specification, and
76 at :age in the data satisfies int? at :age in the specification.

Three datums paired with three predicates. All predicates were satisfied. So valid-scalars? returned true.

Validating partial data

Let's see what happens if we remove Professor Einstein's age from the data but leave the corresponding predicate in the specification.

(valid-scalars? {:first-name "Albert", :last-name "Einstein"}
                {:first-name string?, :last-name string?, :age int?})
;; => true

That result may be surprising. Why doesn't the missing age datum cause a false result? We need to consider Motto #3: Un-paired predicates are ignored. valid-scalars? was able to find two datum+predicate pairs.

"Albert" at :first-name in the data satisfies predicate string? at first-name in the specification.
"Einstein" at :last-name in the data satisfies predicate string? at :last-name in the specification.
Predicate int? at :age in the specification was not paired with an element in the data and was therefore ignored, as Motto #3 informs us.

That may seem kinda broken, but it opens up some powerful capabilities we're about to explore. Later, we'll see how to verify that the age datum actually exists in the data.

Validating complete data, partial specification

What about the other way around? What if our data contains a key-value that does not appear in the specification? Let's add an email entry.

(valid-scalars?
  {:first-name "Albert", :last-name "Einstein", :email "al@princeton.edu"}
  {:first-name string?, :last-name string?})
;; => true

Again, valid-scalars? found two datum+predicate pairs. Both first-name and last-name datums satisfied their corresponding predicates, so the validation returned true. The email datum did not have a corresponding predicate, so, according to Motto #3, it was ignored.

The general idea behind Motto #3 is This element may or may not exist, but if it does exist, it must satisfy this predicate. Taken to its logical conclusion, valid?, within the Speculoos library, conveys the notion Zero invalid results.

Validating complete data, empty specification

We might imagine a scenario where we absolutely do not care about any facet of our data. In that case, our specification would contain exactly zero predicates.

(valid-scalars?
  {:first-name "Albert", :last-name "Einstein", :age 76}
  {})
;; => true

valid-scalars? found zero pairs of datums and predicates. Since there were zero invalids, valid-scalars? returns true.

Validating empty data

Perhaps we've been building up a comprehensive specification for a person's data that includes predicates for a whole slew of possible datums. We need to be able to handle partial data. In other words, not every instance of data we encounter will be complete. The edge case would be zero datums.

(valid-scalars? {}
                {:first-name string?,
                 :last-name string?,
                 :age int?,
                 :address {:street-name string?,
                           :street-number int?,
                           :zip-code int?,
                           :city string?,
                           :state keyword?},
                 :email #"\w@\w"})
;; => true

Not a single one of those predicates was paired with a datum, so there were zero invalid results. Thus, valid-scalars? returns true.

Validating complete data, one invalid datum

In every example we've seen so far, all the datums satisfy the predicate they were paired with. Here's what happens if at least one datum does not satisfy its predicate.

(valid-scalars?
  {:first-name "Albert", :last-name "Einstein", :age "not an integer!"}
  {:first-name string?, :last-name string?, :age int?})
;; => false

String datum "not an integer!" failed to satisfy the int? predicate located at :age in the specification. Therefore, valid-scalars? returned false. Speculoos provides other functions that give more detail about invalid elements, but for simplicity, we'll stick with the true/false results.

Validating presence of a datum

If we wanted to ensure that the data contains a particular key-value, we need to validate the collection itself (Motto #1). Presence of absence of a datum is a property of the collection. First, we'll write a has-age? collection predicate that tests whether the map contains an :age key.

(defn has-age? [m] (contains? m :age))

Then, we insert has-age? into the collection specification. Collection validation operates a little differently from scalar validation, but take my word that this is how to assemble the collection specification for this situation.

{:foo has-age?}

Given this collection specification, has-age? will be paired with the root collection of the data.

Finally, we invoke a completely different function, validate-collections, to validate (Motto #1).

(require '[speculoos.core :refer [valid-collections?]])


(valid-collections? {:first-name "Albert", :last-name "Einstein"}
                    {:foo has-age?})
;; => false

valid-collections? informs us that the map fails to satisfy the has-age? collection predicate. It fails because the map does not contain :age as the collection specification requires.

We'll often want to validate some aspects of both the scalars and the collections, so Speculoos provides a combo function that does a scalar validation, immediately followed by a collection validation, and returns the overall result. The data is the first argument (upper row), the scalar specification is the second argument (middle row), and the collection specification is the third argument (lower row).

(require '[speculoos.core :refer [valid?]])


(valid? {:first-name "Albert", :last-name "Einstein"}
        {:first-name string?, :last-name string?, :age int?}
        {:foo has-age?})
;; => false

The string? scalar predicates at :first-name and :last-name were both satisfied. Scalar predicate int? at :age was ignored because it was un-paired. However, collection predicate has-age? at :foo was not satisfied, so valid? returns false.

Specifying and validating scalar datums separately from specifying and validating collections completely isolates two concerns. The scalar predicates are entirely devoted to testing the properties of the scalars themselves. The collection predicates are devoted to properties of the collections, including the size of the collections, the presence/absence of elements, and relationships between elements. Using both, Speculoos can validate any facet of a heterogeneous, arbitrarily-nested data structure.

When?!

A portion of Maybe Not discusses spec.alpha's issues with optionality. Mr Hickey contends it is a mistake to put optionality into aggregate specifications because doing so destroys a specification's re-usability. An entity that is optional in one context might not be optional in another context.

Speculoos does not suffer from this problem. Because of Motto #3, any predicate that is not paired with a datum is ignored. Any datum that is not paired with predicate is also ignored. Only when a datum is paired with a predicate is the pair considered in the validation result. Separately, if a particular context requires the presence of a datum, we can validate that with a collection validation.

Mr Hickey points out that validating arguments and return values of a function provide a built-in context: the context of the function itself. Speculoos validations themselves carry an inherent context: The context is the validation function combined with the specification, at the moment of invocation, such as we've seen with valid-scalars?.

In addition to being straightforward to compose, Speculoos specifications are extremely flexible because they are plain Clojure data structures. Speculoos specifications can be manipulated using any Clojure tools, including the entire core library or any third-party library. Quite a lot of this flexibility can be demonstrated with merely get, assoc, update, and dissoc.

Let's pretend that a specification someone handed us requires the age datum to be an integer. If Professor Einstein's age is instead a floating-point double, the validation would fail.

(valid-scalars? {:first-name "Albert", :last-name "Einstein", :age 76.0}
                {:first-name string?, :last-name string?, :age int?})
;; => false

76.0 is not an integer.

We can easily relax our specification to accept that the age be any kind of number.

(assoc {:first-name string?, :last-name string?, :age int?} :age number?)
;; => {:age #object [clojure.core$number_QMARK_ 0x38a61c81
;;                   "clojure.core$number_QMARK_@38a61c81"],
;;     :first-name #object [clojure.core$string_QMARK___5494 0xc41709a
;;                          "clojure.core$string_QMARK___5494@c41709a"],
;;     :last-name #object [clojure.core$string_QMARK___5494 0xc41709a
;;                         "clojure.core$string_QMARK___5494@c41709a"]}

With that relaxed specification in hand, that data is now valid.

(valid-scalars? {:first-name "Albert", :last-name "Einstein", :age 76.0}
                (assoc {:first-name string?, :last-name string?, :age int?} :age number?))
;; => true

76.0 satisfies scalar predicate number? which we assoc-ed into the specification on-the-fly.

The original specification is immutable, as it always was. During the validation, we associated a more permissive scalar predicate so that, in this context (while invoking valid-scalars?, with that particular specification), the data is valid.

Now, let's pretend someone handed us a collection specification that requires the presence of the age key-value, but in our little part of the world, our data doesn't have it, and our little machine doesn't need it. Without intervention, collection validation will fail.

(valid-collections? {:first-name "Albert", :last-name "Einstein"}
                    {:foo has-age?})
;; => false

The data (upper row) does not contain a key :age so the data is invalid, according to the specification (lower row) we were handed.

But, our little data processing machine doesn't need Professor Einstein's age, so we can straightforwardly remove that requirement in our context. Here's the altered collection specification.

(dissoc {:foo has-age?} :foo) ;; => {}

If we now use that altered collection specification in the context of our little machine, the data is valid.

(valid-collections? {:first-name "Albert", :last-name "Einstein"}
                    (dissoc {:foo has-age?} :foo))
;; => true

The collection specification no longer contains the has-age? predicate, so the data is declared valid in our context.

Specifications made of plain Clojure data structures absorb every drop of generality, composability, and re-usability of the underlying data structures. They may be passed over the wire, stored in the file system, version controlled, annotated with metadata, and manipulated at will to suit any context.

Replicating specific scenarios from Maybe Not

In this section, we'll explore how Speculoos handles the specific, problematic scenarios presented by Mr Hickey.

Predicates use proper `or`

Speculoos predicates are plain old Clojure functions. When we need to validate an element that may be one of multiple types, the predicates use clojure.core/or, which will inherit all the proper semantics.

Commutative:

(#(or (int? %) (string? %)) 42) ;; => true
(#(or (string? %) (int? %)) 42) ;; => true

Associative:

(#(or (int? %) (or (string? %) (char? %))) 42) ;; => true
(#(or (or (int? %) (string? %)) (char? %)) 42) ;; => true

Distributive:

(#(or (and (int? %) (even? %)) (string? %)) 42) ;; => true

Etc.

`nil`-able

This one's easy: just write Speculoos predicates without nilable.

Namespaced specifications

One of spec.alpha's propositions is that specs are required to be namespace-qualified. Speculoos takes a hands-off approach. Speculoos specifications are plain Clojure data structures that are referenced however we want. Specifications may be a literal, like [int? string? ratio?].

(valid-scalars? [42 "abc" 22/7]
                [int? string? ratio?])
;; => true

Or, we may bind them to a symbol in our current namespace.

(def specification-1 [int? string? ratio?])


(valid-scalars? [42 "abc" 22/7]
                specification-1) ;; => true

(We could also bind them to a symbol in a different namespace; not shown.)

Or, we may gather them into our own bespoke registry.

#'perhaps-so/speculoos-registry

(defonce speculoos-registry (atom {:speculoos/specification-2 [int? string? ratio?]
                                   :speculoos/specification-3 {:first-name string?
                                                               :last-name string?
                                                               :age int?}}))


(valid-scalars? [42 "abc" 22/7]
                (@speculoos-registry :speculoos/specification-2))
;; => true

Cars schema

To follow along precisely, we could split out the make, model, and year concepts into their own named predicate functions, but for brevity, I'll stuff them directly into our car scalar specification.

(def car-scalar-specification
  {:make string?, :model string?, :year #(and (int? %) (>= % 1885))})

At this point, we're not stating anything definitive about presence or absence of an element. A scalar specification says, for each scalar predicate, This element may or may not exist, but if it does, the element must satisfy this predicate. Declaring that :make is a string is completely separate from declaring that our car data must contain a :make value.

If we want to require that our car data contains a :make entry, we declare that requirement in a collection specification.

(def car-collection-specification {:foo #(contains? % :make)})

Unless explicitly required in a collection specification, Speculoos treats all scalars as optional. So we don't have to say anything about :model or :year.

Let's validate with all the specified values.

(valid? {:make "Acme Motor Cars", :model "Type 1", :year 1905}
        car-scalar-specification
        car-collection-specification)
;; => true

The values we supplied for :make, :model, and year all satisfied their respective scalar predicates. Furthermore, the car map itself satisfied the collection specification's requirement that the map contain a key :make.

Now, let's validate some car data with partial information: :model and :year values are missing.

(valid? {:make "Acme Motor Cars"}
        car-scalar-specification
        car-collection-specification)
;; => true

{:make "Acme Motor Cars"} is valid car data because :make satisfies its scalar predicate and the map itself contains the only key we required in the collection specification. :model and :year are implicitly optional because we did not require their existence in the collection specification.

What if we have extra information? Let's validate data about an early 1900s car with a completely anachronistic computer chip.

(valid? {:make "Acme Motor Cars", :year 1905, :cpu "Intel Pentium"}
        car-scalar-specification
        car-collection-specification)
;; => true

Again, this car data is valid, because we did not specify any property relating to :cpu, so the validation ignored that datum. All the other existing datums satisfied their corresponding predicates.

What if we neglect to include the :make element?

(valid? {:model "Type 1", :year 1905}
        car-scalar-specification
        car-collection-specification)
;; => false

Finally, we run afoul of our collection specification: Our car data lacks a :make element, which our collection specification explicitly requires.

What if we're in a different context, and suddenly we absolutely must have a :year element, too? Right then and there, we can augment the collection specification, because it's a plain Clojure data structure. And we know how to associate items on-the-fly into a map.

(assoc car-collection-specification :bar #(contains? % :year))
;; => {:foo #(contains? % :make), :bar #(contains? % :year)}

So in this new context, we use that new collection specification with tighter requirements.

(valid? {:make "Acme Motor Cars", :model "Type 1"}
        car-scalar-specification
        (assoc car-collection-specification :bar #(contains? % :year)))
;; => false

Now, the absence of :year element of our car data is no longer ignored. This car data fails to satisfy one of its collection predicates that require the presence of a :year entry.

Note that specifying the values of the scalars themselves (in a Speculoos scalar specification), is completely orthogonal to requiring their presence (which we declare in a Speculoos collection specification). By splitting the two concerns, the specifications become straightforwardly re-usable. We are free to specify any number of properties of the car data that may or may not exist. Then, in a particular context, we adjust our collection specification to require the particular group of elements that we need for that particular context.

Speculoos specifications shouldn't proliferate uncontrollably because, as plain Clojure data structures, they're readily manipulable, on-the-fly, with assoc and friends.

Symmetric request/response schemas

Give me a partially filled-in form, and I will give you back a more filled-in form. No problem for Speculoos to validate that scenario with a single scalar specification. Because un-paired predicates are ignored, we can simply use the same scalar specification to validate both the before data and the after data.

Let's pretend we query a service with an ID, and the service returns that ID and the associated name and phone number. That would be a straightforward scalar specification.

(def one-spec {:ID int?, :name string?, :phone int?})

Before we submit our request to the service, let's validate the partially filled-in form.

(valid-scalars? {:ID 99} one-spec) ;; => true

valid-scalars? only considers pairs of datum+predicates. Our partially filled-in request only has one datum that is paired with a predicate: 99 at :ID is paired with int? at :ID in the scalar specification. 99 satisfies its paired scalar predicate int?, so valid-scalars? returns true.

Now that we've validated our request, we send it off to the service, which returns {:ID 99 :name "Sherlock Holmes" :phone 123456789}. We can validate the response with the exact same scalar specification.

(valid-scalars? {:ID 99, :name "Sherlock Holmes", :phone 12345678} one-spec)
;; => true

During this invocation, valid-scalars? encountered three datum+predicate pairs, and each datum satisfied its corresponding scalar predicate, so the validation returns true.

What if we submit a query that causes the service to emit garbage data like {:ID 0 :name \z :phone \q}? Let's validate that.

(valid-scalars? {:ID 99, :name \z, :phone \q} one-spec) ;; => false

Same scalar specification, but since the datums do not satisfy their scalar predicates, the service's response does not satisfy the specification.

One specification is sufficient to validate the data at each step. The specification is re-used, and there's one authoritative description of what an ID/name/phone aggregate looks like.

Information-building pipelines

An information-building pipeline is merely repeated application of the principles embodied in the request/response pattern we discussed earlier. A singular scalar specification can describe all the steps of a serially aggregating data structure.

Let's pretend our cupcake processing pipeline accepts an accumulating map, and adds a new quantity based on an ingredient we pass alongside. It might look something like this.

empty bowl
flour → 150 grams
eggs → 2
sugar → 130 grams
butter → 60 grams
milk → 1/8 liter

Our pipeline constructs the accumulating map in six steps like this.

{}
{:flour 150.0}
{:flour 150.0 :eggs 2}
{:flour 150.0 :eggs 2 :sugar 130.0}
{:flour 150.0 :eggs 2 :sugar 130.0 :butter 60.0}
{:flour 150.0 :eggs 2 :sugar 130.0 :butter 60.0 :milk 1/8}

We can write one single scalar specification that will validate the result of each of those six steps.

(def cupcake-spec
  {:flour double?, :eggs int?, :sugar double?, :butter double?, :milk ratio?})

Now, we can validate the data at each step.

(valid-scalars? {}
                cupcake-spec) ;; => true

(valid-scalars? {:flour 150.0}
                cupcake-spec) ;; => true

(valid-scalars? {:flour 150.0, :eggs 2}
                cupcake-spec) ;; => true

(valid-scalars? {:flour 150.0, :eggs 2, :sugar 130.0}
                cupcake-spec)
;; => true

(valid-scalars? {:flour 150.0, :eggs 2, :sugar 130.0, :butter 60.0}
                cupcake-spec)
;; => true

(valid-scalars? {:flour 150.0, :eggs 2, :sugar 130.0, :butter 60.0, :milk 1/8}
                cupcake-spec)
;; => true

Notice: cupcake-spec remained the same for each of the six validations as the pipeline added more and more elements. Speculoos' policy of ignoring un-paired predicates offers us the ability to specify the final shape at the outset, and the validation only considers the elements present at the moment of invocation.

Nested schemas

Speculoos was designed from the outset to validate any heterogeneous, arbitrarily-nested data structure. Mr Hickey imagines a data structure something like this.

{:a 42
 :b "abc"
 :c [\x \y \z]
 :d ['foo 'bar 'baz]}

We can immediately compose a specification for that data. One trick is to take advantage of the fact that Speculoos specifications mimic the shape of the data (Motto #2). So first, we copy-paste the data, and delete the scalars.

{:a    :b     :c [       ] :d [        ]}

Then, we insert our predicates, one predicate for each scalar.

{:a int? :b string? :c [char? char? char?] :d [symbol? symbol? symbol?]}

Finally, we validate.

(valid-scalars?
  {:a 42, :b "abc", :c [\x \y \z], :d ['foo 'bar 'baz]}
  {:a int?, :b string?, :c [char? char? char?], :d [symbol? symbol? symbol?]})
;; => true

valid-scalars? systematically marches through the data and specification, searching for pairs of scalars and predicates. In this case, it finds pairs at keys :a and :b, and dives down into the nested vectors at keys :c and :d. So scalar \x is paired with predicate char?, scalar 'foo is paired with predicate symbol?, etc. All the scalars satisfy their corresponding predicates, so the validation returns true.

We could also compose an equivalent scalar specification from pre-defined subcomponents. Consider this.

(def three-chars? [char? char? char?])
(def three-syms? [symbol? symbol? symbol?])


(valid-scalars? {:a 42, :b "abc", :c [\x \y \z], :d ['foo 'bar 'baz]}
                {:a int?, :b string?, :c three-chars?, :d three-syms?})
;; => true

Regular old Clojure composition in action. The scalar specification refers to three-chars? at its key :c and refers to three-syms? at its key :d. We can thus mix and match with impunity to compose our specifications.

A riff on that tune is to extract some selection of our data and validate it against a smaller specification. Pretend we only care about validating the three-element vector at :c. We've got tools that can pull that vector out.

(get {:a 42, :b "abc", :c [\x \y \z], :d ['foo 'bar 'baz]} :c) ;; => [\x \y \z]

And we've already written a specification for that extracted vector, three-chars?, so we can immediately validate.

(valid-scalars? (get {:a 42, :b "abc", :c [\x \y \z], :d ['foo 'bar 'baz]} :c)
                three-chars?)
;; => true

This invocation used get to extract the vector at :c and validated it against predicate three-chars?.

Alternatively, we could leverage the fact that un-paired datums are ignored, and specify only that nested vector.

(valid-scalars? {:a 42, :b "abc", :c [\x \y \z], :d ['foo 'bar 'baz]}
                {:c three-chars?})
;; => true

We performed a validation on only a selected slice of data because valid-scalars? applied only the three char? scalar predicates to the contents of the nested vector at :c.

Movie times & placing orders

Mr Hickey's next example extends the discussion of validating Professor Einstein's data from earlier. First, we'll write some bottom-level specifications.

(def street string?)
(def city string?)
(def state keyword?)
(def zip int?)

Then, we aggregate them into a specification for an address.

(def address {:street street, :city city, :state state, :zip zip})

Next, we write some more bottom-level specifications.

(def id int?)
(def first-name string?)
(def last-name string?)

Finally, we aggregate those specifications into a user specification, including the address aggregate from before.

(def user
  {:id id, :first-name first-name, :last-name last-name, :address address})

Speculoos provides several utilities for validating function arguments and return values, but to avoid introducing a new utility, we'll stick with valid-scalars? and I'll ask that you trust me that the function validation operates substantially the same way.

If we imagine that a function get-movie-times expects an ID and a zip, we could validate that slice of data.

(valid-scalars?
  {:id 101, :address {:zip 90210}}
  {:id id, :first-name first-name, :last-name last-name, :address address})
;; => true

In the context of a different function, place-order, we might want to validate a first name, last name, and the full address. Validating that slice of data would look like this.

(valid-scalars? {:first-name "Helen",
                 :last-name "tis Troías",
                 :address {:street "Equine Avenue",
                           :city "Sparta",
                           :state :LCNIA,
                           :zip 54321}}
                {:id id,
                 :first-name first-name,
                 :last-name last-name,
                 :address address})
;; => true

Exact same specification, user, in both cases, but this time, a different slice of data was compared to specification. Because Speculoos only validates using predicates that are paired with scalars, the extra, un-paired predicates (in this example, :id) in scalar specification user have no effect.

Also note that the data is a heterogeneous, nested data structure (Mr Hickey calls it a 'tree'), and because Speculoos specifications mimic the shape of the data, the user scalar specification is also a tree. Speculoos can handle any depth of nesting, with any of Clojure's data structures (vectors, lists, sequences, maps, and sets).

No requirements

Speculoos will happily validate data with an empty specification.

(valid-scalars? {:sheep #{"Fred" "Ethel"}, :helicopters 1}
                {})
;; => true

Validating with an empty specification will always return true. That behavior is governed by ignoring un-paired scalars (i.e., there are no predicates to pair with), and zero un-satisfied predicates is considered valid. Speculoos is 'open' in the sense that Mr Hickey discusses: Extra information is okay. Speculoos merely ignores it if it isn't paired with a predicate.

Speculoos can generate valid test data if we supply it with a scalar specification.

(require '[speculoos.utility :refer [exercise]])


(exercise {:sheep #{"Fred" "Ethel" "Lucy" "Ricky" "Little Ricky"},
           :helicopters pos-int?}
          5)
;; => ([{:helicopters 24, :sheep "Ethel"} true]
;;     [{:helicopters 30, :sheep "Ricky"} true]
;;     [{:helicopters 26, :sheep "Ethel"} true]
;;     [{:helicopters 11, :sheep "Little Ricky"} true]
;;     [{:helicopters 17, :sheep "Lucy"} true])

Programmatically manipulating specifications

We discussed this earlier. Speculoos specifications are plain Clojure data structures containing plain predicate functions. Slice and dice them however we want.

Extract a slice of a specification, perhaps just the address.

(get {:id int?,
      :first-name string?,
      :last-name string?,
      :address {:street string?,
                :city string?,
                :zip int?,
                :state keyword?}}
     :address)
;; => {:city
;;       #object
;;        [clojure.core$string_QMARK___5494
;;         0xc41709a
;;         "clojure.core$string_QMARK___5494@c41709a"],
;;     :state
;;       #object
;;        [clojure.core$keyword_QMARK_ 0x4f94547a
;;         "clojure.core$keyword_QMARK_@4f94547a"],
;;     :street
;;       #object
;;        [clojure.core$string_QMARK___5494
;;         0xc41709a
;;         "clojure.core$string_QMARK___5494@c41709a"],
;;     :zip #object
;;           [clojure.core$int_QMARK_ 0x1545b9da
;;            "clojure.core$int_QMARK_@1545b9da"]}

Alter a portion of a specification, perhaps by tightening the requirements of the ID.

(assoc {:id int?,
        :first-name string?,
        :last-name string?,
        :address {:street string?,
                  :city string?,
                  :zip int?,
                  :state keyword?}}
  :id even?)
;; => {:address
;;       {:city
;;          #object
;;           [clojure.core$string_QMARK___5494
;;            0xc41709a
;;            "clojure.core$string_QMARK___5494@c41709a"],
;;        :state
;;          #object
;;           [clojure.core$keyword_QMARK_
;;            0x4f94547a
;;            "clojure.core$keyword_QMARK_@4f94547a"],
;;        :street
;;          #object
;;           [clojure.core$string_QMARK___5494
;;            0xc41709a
;;            "clojure.core$string_QMARK___5494@c41709a"],
;;        :zip
;;          #object
;;           [clojure.core$int_QMARK_ 0x1545b9da
;;            "clojure.core$int_QMARK_@1545b9da"]},
;;     :first-name
;;       #object
;;        [clojure.core$string_QMARK___5494
;;         0xc41709a
;;         "clojure.core$string_QMARK___5494@c41709a"],
;;     :id #object
;;          [clojure.core$even_QMARK_ 0x71713254
;;           "clojure.core$even_QMARK_@71713254"],
;;     :last-name
;;       #object
;;        [clojure.core$string_QMARK___5494
;;         0xc41709a
;;         "clojure.core$string_QMARK___5494@c41709a"]}

We could, on-the-fly, require :id to be a positive integer by invoking valid-scalars? with that assoc-ed specification. The original specification is immutable as always, and remains unchanged. But at that moment of validation, the requirements were tightened.

No need to write a macro. Just manipulate Clojure data with our favorite utilities.

Function validation

Speculoos has an entire namespace dedicated to validating function arguments and return values. Function validation follows all the same principles we've been discussing about validating data. It's a lengthy topic, so I'll refer to the documentation on the subject.

Nail down everything!

I hope at this point I've made a convincing case that Speculoos is open and permissive: Only specify what we're interested in, and Speculoos will ignore the rest. There is no requirement that we describe every possible facet of our data. Whatever small amount we do specify, can be used to validate our data, and generate test samples, etc.

Final thoughts

It's fortunate that Speculoos' implementation details combined with a few policy decisions resulted in being able to address most all of Maybe Not's concerns. I don't claim that Speculoos is the only solution to these issues, but that the principles under which Speculoos operates provides one possible solution.

Let me know what you think.