Issue #14187 has been updated by mame (Yusuke Endoh).


naruse (Yui NARUSE) wrote:
> Casual users really needs to run test-all and rubyspec?

The short answer is "I think so".

I agree that there is many "levels" of test suites: easy tests for casual users to verify their installation, daily tests for developers to verify their changes, exhaustive tests for CI to verify abnormal handling code like out of memory, etc.  However, there is currently no consensus about the relationship between the levels and the test suites.  If you reorganize (or split) the test suites into each level, it would be a good idea to assign `make test` only to "easy test for casual users".  But currently there is no such levels.  To avoid missing an issue, it would be good for casual users to run all, I think.

By the way, we cannot use the name `test-basic` because there has been already `make test-basic` to run `basictest/`.  Is there any other idea?

----------------------------------------
Feature #14187: `make test` and `make check` to run all test suites
https://bugs.ruby-lang.org/issues/14187#change-72517

* Author: mame (Yusuke Endoh)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: next minor
----------------------------------------
Currently, MRI has many test suites: "`bootstraptest/`", "`KNOWNBUGS.rb`", "`basictest/`", "`test/`", and "`spec/`".  And, the target name design of `make` is really complicated:

* `make test` runs the first three suites
* `make test-all` runs only `test/`
* `make check` runs all the suites except `spec/`
* `make exam` runs all the suites

I think when a casual user build and install ruby, s/he will expect `make test` or `make check` to run all recommended test suites because many other projects uses them as the semantics.

Thus, how about changing `make test` and `make check` to run all test suites?

I think this change will affect some ruby development tools including `rubyci` (`chkbuild`), `mswin-build`, and `ruby-loco`.  @hsbt, @usa, @MSP-Greg, what do you think?

```diff
diff --git a/common.mk b/common.mk
index 80001b481c..ba5fffdf4b 100644
--- a/common.mk
+++ b/common.mk
@@ -634,10 +634,6 @@ clean-spec: PHONY
 	-$(Q) $(RM) $(RUBYSPEC_CAPIEXT)/*.$(OBJEXT) $(RUBYSPEC_CAPIEXT)/*.$(DLEXT)
 	-$(Q) $(RMDIRS) $(RUBYSPEC_CAPIEXT) 2> $(NULL) || exit 0
 
-check: main test test-testframework test-almost
-	$(ECHO) check succeeded
-check-ruby: test test-ruby
-
 fake: $(CROSS_COMPILING)-fake
 yes-fake: $(arch)-fake.rb $(RBCONFIG) PHONY
 no-fake -fake: PHONY
@@ -660,6 +656,8 @@ no-btest-ruby: PHONY
 yes-btest-ruby: prog PHONY
 	$(Q)$(exec) $(RUNRUBY) "$(srcdir)/bootstraptest/runner.rb" --ruby="$(PROGRAM) -I$(srcdir)/lib $(RUN_OPTS)" -q $(OPTS) $(TESTOPTS)
 
+test-bootstrap: btest-ruby
+
 test-basic: $(TEST_RUNNABLE)-test-basic
 no-test-basic: PHONY
 yes-test-basic: prog PHONY
@@ -677,7 +675,6 @@ yes-test-testframework: prog PHONY
 no-test-testframework: PHONY
 
 test-sample: test-basic # backward compatibility for mswin-build
-test: btest-ruby test-knownbug test-basic
 
 # $ make test-all TESTOPTS="--help" displays more detail
 # for example, make test-all TESTOPTS="-j2 -v -n test-name -- test-file-name"
@@ -755,7 +752,7 @@ $(ENC_MK): $(srcdir)/enc/make_encmake.rb $(srcdir)/enc/Makefile.in $(srcdir)/enc
 .PHONY: clean clean-ext clean-local clean-enc clean-golf clean-rdoc clean-html clean-extout
 .PHONY: distclean distclean-ext distclean-local distclean-enc distclean-golf distclean-extout
 .PHONY: realclean realclean-ext realclean-local realclean-enc realclean-golf realclean-extout
-.PHONY: check test test-all btest btest-ruby test-basic test-knownbug
+.PHONY: exam check test test-all test-bootstrap btest btest-ruby test-basic test-knownbug
 .PHONY: run runruby parse benchmark benchmark-each tbench gdb gdb-ruby
 .PHONY: update-mspec update-rubyspec test-rubyspec test-spec
 .PHONY: touch-unicode-files
@@ -1322,9 +1319,12 @@ info-arch: PHONY
 change: PHONY
 	$(BASERUBY) -C "$(srcdir)" ./tool/change_maker.rb $(CHANGES) > change.log
 
-exam: check test-spec
+test: main test-bootstrap test-knownbug test-basic test-testframework test-almost test-spec
+	$(ECHO) check succeeded
+check: test
+exam: test
 
-love: sudo-precheck up all test install exam
+love: sudo-precheck up all test install
 	@echo love is all you need
 
 great: exam
@@ -1356,11 +1356,11 @@ help: PHONY
 	"  runruby:             runs test.rb by ruby you just built" \
 	"  gdb:                 runs test.rb by miniruby under gdb" \
 	"  gdb-ruby:            runs test.rb by ruby under gdb" \
-	"  check:               equals make test test-all" \
-	"  exam:                equals make check test-spec" \
-	"  test:                ruby core tests" \
-	"  test-all:            all ruby tests [TESTOPTS=-j4 TESTS=<test files>]" \
-	"  test-spec:           run the Ruby spec suite" \
+	"  test:                ruby all test suites" \
+	"  check:               equals make test" \
+	"  exam:                equals make test" \
+	"  test-all:            run `test/` suite [TESTOPTS=-j4 TESTS=<test files>]" \
+	"  test-spec:           run the Ruby spec suite (in `spec/`)" \
 	"  test-rubyspec:       same as test-spec" \
 	"  test-bundler:        run the Bundler spec" \
 	"  test-bundled-gems:   run the test suite of bundled gems" \
```



-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:ruby-core-request / ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>