Age | Commit message (Collapse) | Author |
|
Add the missing EC event codes in elog.h and correct the event code value for
RECOVERY_HWREINIT.
Change-Id: If9fb319cce1e4acce4b3d7c3a39365986856a9b4
Signed-off-by: Furquan Shaikh <furquan@chromium.org>
Reviewed-on: https://review.coreboot.org/18693
Tested-by: build bot (Jenkins)
Reviewed-by: Duncan Laurie <dlaurie@chromium.org>
Reviewed-by: Aaron Durbin <adurbin@chromium.org>
Reviewed-by: Paul Menzel <paulepanter@users.sourceforge.net>
|
|
The current elog implementation has two event types defined for 0xa7,
apparently the result of divergent coreboot trees on chromium where
some events were added to ARM systems but not upstreamed until later.
Fix this by moving ELOG_TYPE_THERM_TRIP to be 0xab, since the current
elog parsing code in chromium is using ELOG_TYPE_SLEEP for 0xa7.
BUG=b:35977516
TEST=check for proper "CPU Thermal Trip" event when investigating a
device that is unexpectedly powering down.
Change-Id: Idfa9b2322527803097f4f19f7930ccbdf2eccf35
Signed-off-by: Duncan Laurie <dlaurie@chromium.org>
Reviewed-on: https://review.coreboot.org/18579
Tested-by: build bot (Jenkins)
Reviewed-by: Furquan Shaikh <furquan@google.com>
Reviewed-by: Paul Menzel <paulepanter@users.sourceforge.net>
Reviewed-by: Aaron Durbin <adurbin@chromium.org>
|
|
Piggy-back on existing MRC cache infrastructure to store variable MRC data.
Only one set of data can be valid at given point of time. Currently this
magically happens because region alignment is forced to 0x1000 and region
itself is of the same size. This needs to be somehow programmatically
enforced.
Change-Id: I8a660d356ca760b8ff9907396fb9b34cb16cf1db
Signed-off-by: Andrey Petrov <andrey.petrov@intel.com>
Reviewed-on: https://review.coreboot.org/17320
Tested-by: build bot (Jenkins)
Reviewed-by: Aaron Durbin <adurbin@chromium.org>
|
|
BUG=chrome-os-partner:59352
BRANCH=None
TEST=Compiles successfully
Change-Id: Idf2d377bf4709ea25616adfbde55f39798c0cd39
Signed-off-by: Furquan Shaikh <furquan@chromium.org>
Reviewed-on: https://review.coreboot.org/17393
Tested-by: build bot (Jenkins)
Reviewed-by: Aaron Durbin <adurbin@chromium.org>
|
|
1. Re-factor MRC cache driver to properly select RW_MRC_CACHE or
RECOVERY_MRC_CACHE based on the boot mode.
- If normal mode boot, use RW_MRC_CACHE, if available.
- If recovery mode boot:
- Retrain memory if RECOVERY_MRC_CACHE not present, or recovery is
requested explicity with retrain memory request.
- Use RECOVERY_MRC_CACHE otherwise.
2. Protect RW and RECOVERY mrc caches in recovery and non-recovery boot
modes. Check if both are present under one unified region and protect
that region as a whole. Else try protecting individual regions.
3. Update training data in appropriate cache:
- Use RW_MRC_CACHE if normal mode.
- Use RECOVERY_MRC_CACHE if present in recovery mode. Else use
RW_MRC_CACHE.
4. Add proper debug logs to indicate which training data cache is used
at any point.
BUG=chrome-os-partner:59352
BRANCH=None
TEST=Verified that correct cache is used in both normal and recovery
mode on reef.
Change-Id: Ie79737a1450bd1ff71543e44a5a3e16950e70fb3
Signed-off-by: Furquan Shaikh <furquan@chromium.org>
Reviewed-on: https://review.coreboot.org/17242
Tested-by: build bot (Jenkins)
Reviewed-by: Aaron Durbin <adurbin@chromium.org>
|
|
Log when the MRC cache is attempted to be updated with status
of success or failure. Just one slot is supported currently
which is deemed 'normal'. This is because there are more slots
anticipated in the future.
BUG=chrome-os-partner:59395
Change-Id: I0f81458325697aff9924cc359a4173e0d35da5da
Signed-off-by: Aaron Durbin <adurbin@chromium.org>
Reviewed-on: https://review.coreboot.org/17231
Reviewed-by: Furquan Shaikh <furquan@google.com>
Tested-by: build bot (Jenkins)
|
|
Some events were added in other places, but coreboot's
elog namespace wasn't updated. As such there's a collision
with the thermtrip event. This change at least updates the
elog information to reflect potential event type uage.
BUG=chrome-os-partner:59395
Change-Id: Ib82e2b65ef7d34e260b7d7450174aee7537b69f6
Signed-off-by: Aaron Durbin <adurbin@chromium.org>
Reviewed-on: https://review.coreboot.org/17230
Tested-by: build bot (Jenkins)
Reviewed-by: Furquan Shaikh <furquan@google.com>
|
|
When CONFIG_ELOG isn't used default empty inline functions are
provided, however the elog_add_event_raw() had the wrong type
signature. Fix that.
BUG=chrome-os-partner:59395
Change-Id: Iaee68440bbafc1e91c88a7b03e283fc3e72de0a3
Signed-off-by: Aaron Durbin <adurbin@chromium.org>
Reviewed-on: https://review.coreboot.org/17232
Tested-by: build bot (Jenkins)
Reviewed-by: Furquan Shaikh <furquan@google.com>
|
|
Instead of relying on global state to determine if an error
occurred provide the ability to know if an add or shrink
operation is successful. Now the call chains report the
error back up the stack and out to the callers.
BUG=chrome-os-partner:55932
Change-Id: Id4ed4d93e331f1bf16e038df69ef067446d00102
Signed-off-by: Aaron Durbin <adurbin@chromium.org>
Reviewed-on: https://review.coreboot.org/16104
Tested-by: build bot (Jenkins)
Reviewed-by: Furquan Shaikh <furquan@google.com>
|
|
There is no need to add guards around boot_count_* functions since the
static definition of boot_count_read is anyways unused.
BUG=chrome-os-partner:55473
Change-Id: I553277cdc09a8af420ecf7caefcb59bc3dcb28f1
Signed-off-by: Furquan Shaikh <furquan@google.com>
Reviewed-on: https://review.coreboot.org/15997
Tested-by: build bot (Jenkins)
Reviewed-by: Patrick Georgi <pgeorgi@google.com>
|
|
Provide default handler for some SMI events. Provide the framework for
extracting data from SMM Save State area for processors with SMM revision
30100 and 30101.
The SOC specific code should initialize southbridge_smi with event
handlers. For SMM Save state handling, SOC code should implement
get_smm_save_state_ops which initializes the SOC specific ops for SMM Save
State handling.
Change-Id: I0aefb6dbb2b1cac5961f9e43f4752b5929235df3
Signed-off-by: Hannah Williams <hannah.williams@intel.com>
Reviewed-on: https://review.coreboot.org/14615
Tested-by: build bot (Jenkins)
Reviewed-by: Aaron Durbin <adurbin@chromium.org>
|
|
It encourages users from writing to the FSF without giving an address.
Linux also prefers to drop that and their checkpatch.pl (that we
imported) looks out for that.
This is the result of util/scripts/no-fsf-addresses.sh with no further
editing.
Change-Id: Ie96faea295fe001911d77dbc51e9a6789558fbd6
Signed-off-by: Patrick Georgi <pgeorgi@chromium.org>
Reviewed-on: http://review.coreboot.org/11888
Tested-by: build bot (Jenkins)
Reviewed-by: Alexandru Gagniuc <mr.nuke.me@gmail.com>
Reviewed-by: Ronald G. Minnich <rminnich@gmail.com>
|
|
As per discussion with lawyers[tm], it's not a good idea to
shorten the license header too much - not for legal reasons
but because there are tools that look for them, and giving
them a standard pattern simplifies things.
However, we got confirmation that we don't have to update
every file ever added to coreboot whenever the FSF gets a
new lease, but can drop the address instead.
util/kconfig is excluded because that's imported code that
we may want to synchronize every now and then.
$ find * -type f -exec sed -i "s:Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, *MA[, ]*02110-1301[, ]*USA:Foundation, Inc.:" {} +
$ find * -type f -exec sed -i "s:Foundation, Inc., 51 Franklin Street, Suite 500, Boston, MA 02110-1335, USA:Foundation, Inc.:" {} +
$ find * -type f -exec sed -i "s:Foundation, Inc., 59 Temple Place[-, ]*Suite 330, Boston, MA *02111-1307[, ]*USA:Foundation, Inc.:" {} +
$ find * -type f -exec sed -i "s:Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.:Foundation, Inc.:" {} +
$ find * -type f
-a \! -name \*.patch \
-a \! -name \*_shipped \
-a \! -name LICENSE_GPL \
-a \! -name LGPL.txt \
-a \! -name COPYING \
-a \! -name DISCLAIMER \
-exec sed -i "/Foundation, Inc./ N;s:Foundation, Inc.* USA\.* *:Foundation, Inc. :;s:Foundation, Inc. $:Foundation, Inc.:" {} +
Change-Id: Icc968a5a5f3a5df8d32b940f9cdb35350654bef9
Signed-off-by: Patrick Georgi <pgeorgi@chromium.org>
Reviewed-on: http://review.coreboot.org/9233
Tested-by: build bot (Jenkins)
Reviewed-by: Vladimir Serbinenko <phcoder@gmail.com>
|
|
Provide elog stub functions so eventlog support can be omitted
without littering code with "#if CONFIG_ELOG".
This makes it so coreboot can be built without eventlog support for
these platforms for debugging purposes.
BUG=none
BRANCH=none
TEST=compiled for Nyan and Rambi with CONFIG_ELOG unset
Signed-off-by: David Hendricks <dhendrix@chromium.org>
Original-Change-Id: Ibf56d29a09234068773378f99ad9bffd5480dc9c
Original-Reviewed-on: https://chromium-review.googlesource.com/198647
Original-Reviewed-by: Julius Werner <jwerner@chromium.org>
Original-Reviewed-by: Tom Warren <twarren@nvidia.com>
Original-Commit-Queue: David Hendricks <dhendrix@chromium.org>
Original-Tested-by: David Hendricks <dhendrix@chromium.org>
(cherry picked from commit 8e83dd460647972c4f46c19f8dc3d3ad7baeb550)
Signed-off-by: Marc Jones <marc.jones@se-eng.com>
Change-Id: I3c0803ceb7a1c06da717416c42b6b7730c029ed0
Reviewed-on: http://review.coreboot.org/7901
Reviewed-by: Edward O'Callaghan <eocallaghan@alterapraxis.com>
Tested-by: build bot (Jenkins)
Reviewed-by: David Hendricks <dhendrix@chromium.org>
|
|
This severs a dependency the eventlog code has on initializing
chipset/SoC SPI controller. Currently elog_init() calls spi_init()
as a catch-all. This worked for x86 since the SPI controller is only
used for one thing on existing platforms. As we add eventlogging
support to non-x86 platforms we need to consider the more generalized
case where the assumptions about how SPI works on x86 are no longer
valid.
BUG=none
BRANCH=none
Signed-off-by: David Hendricks <dhendrix@chromium.org>
TEST=built and booted on Link, Beltino and Rambi. See below for
"mosys eventlog list" output on Link showing boot and suspend/resume
events (including lid close/open) added successfully.
localhost ~ # mosys eventlog list
0 | 2014-04-14 13:52:44 | Log area cleared | 4096
1 | 2014-04-14 13:52:44 | System boot | 50
2 | 2014-04-14 13:52:44 | EC Event | Power Button
3 | 2014-04-14 13:52:44 | SUS Power Fail
4 | 2014-04-14 13:52:44 | System Reset
5 | 2014-04-14 13:52:44 | ACPI Wake | S5
6 | 2014-04-14 13:53:25 | ACPI Enter | S3
7 | 2014-04-14 13:53:35 | ACPI Wake | S3
8 | 2014-04-14 13:53:35 | Wake Source | RTC Alarm | 0
9 | 2014-04-14 13:53:49 | ACPI Enter | S3
10 | 2014-04-14 13:54:00 | EC Event | Lid Open
11 | 2014-04-14 13:54:00 | ACPI Wake | S3
12 | 2014-04-14 13:54:00 | Wake Source | GPIO | 15
Original-Change-Id: I26e25c0a856f7b8db5ab6b8e7e1acae291d2eadc
Original-Reviewed-on: https://chromium-review.googlesource.com/194526
Original-Reviewed-by: David Hendricks <dhendrix@chromium.org>
Original-Commit-Queue: David Hendricks <dhendrix@chromium.org>
Original-Tested-by: David Hendricks <dhendrix@chromium.org>
(cherry picked from commit 2971d20b6ebdd9803b05ccbbaeefe1bde1a21af4)
Signed-off-by: Marc Jones <marc.jones@se-eng.com>
Change-Id: Ia5f2913fd8e4fee6e741e6d1e39d32bb86525cb3
Reviewed-on: http://review.coreboot.org/7831
Tested-by: build bot (Jenkins)
Reviewed-by: Stefan Reinauer <stefan.reinauer@coreboot.org>
|
|
There is a status bit for this event in most intel chipsets that
we can read and report. Start by adding the new event type.
Change-Id: Ib06411e3b87a1d069fb469943dd445bee6c1291f
Signed-off-by: Duncan Laurie <dlaurie@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/199370
Reviewed-by: Aaron Durbin <adurbin@chromium.org>
(cherry picked from commit 386a06170ec5afb31d0fe93ace3afbaab897a598)
Signed-off-by: Isaac Christensen <isaac.christensen@se-eng.com>
Reviewed-on: http://review.coreboot.org/7004
Tested-by: build bot (Jenkins)
Reviewed-by: Patrick Georgi <patrick@georgi-clan.de>
|
|
Fix compilation. Relying on the pre-processor to condition an if
statement will lead to warnings of implicitly defined functions. To
solve this dilemma add symbols to resolve to at compile time.
Change-Id: Id0117528c5579cc1dec750a8a17a76fab4314b3f
Signed-off-by: Edward O'Callaghan <eocallaghan@alterapraxis.com>
Reviewed-on: http://review.coreboot.org/5504
Tested-by: build bot (Jenkins)
Reviewed-by: Alexandru Gagniuc <mr.nuke.me@gmail.com>
|
|
This can be used to indicate sub-state within a POST
code range which can assist in debugging BIOS hangs.
For example this can be used to indicate which device
is about to be initialized so if the system hangs
while talking to that device it can be identified.
Change-Id: I2f8155155f09fe9e242ebb7204f0b5cba3a1fa1e
Signed-off-by: Duncan Laurie <dlaurie@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/58104
Reviewed-on: http://review.coreboot.org/4229
Tested-by: build bot (Jenkins)
Reviewed-by: Alexandru Gagniuc <mr.nuke.me@gmail.com>
|
|
Change-Id: Ida98f81b1ac1f6b3ba16c0b98e5c64756606fd58
Signed-off-by: Marc Jones <marc.jones@se-eng.com>
Reviewed-on: https://gerrit.chromium.org/gerrit/48318
Reviewed-by: Stefan Reinauer <reinauer@google.com>
Tested-by: Stefan Reinauer <reinauer@google.com>
Reviewed-on: http://review.coreboot.org/4126
Reviewed-by: Paul Menzel <paulepanter@users.sourceforge.net>
Reviewed-by: Ronald G. Minnich <rminnich@gmail.com>
Tested-by: build bot (Jenkins)
|
|
The EC saves its last "shutdown reason" for the system in EC RAM
that we can read back and log on boot.
The decode for the "reason" field will be added to mosys.
Change-Id: I834d39122e45262ef8e7ba59201accbee5857aac
Signed-off-by: Duncan Laurie <dlaurie@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/48323
Reviewed-by: David James <davidjames@chromium.org>
Reviewed-by: David Hendricks <dhendrix@chromium.org>
Commit-Queue: Stefan Reinauer <reinauer@google.com>
Tested-by: Stefan Reinauer <reinauer@google.com>
Reviewed-on: http://review.coreboot.org/4127
Tested-by: build bot (Jenkins)
|
|
This is updated to handle LynxPoint-H and LynxPoint-LP
and a new wake event is added for the power button.
Boot, suspend/resume, reboot, etc on WTM2
and then check the event log to see if expected events
have been added.
Change-Id: I15cbc3901d81f4fd77cc04de37ff5fa048f9d3e8
Signed-off-by: Duncan Laurie <dlaurie@chromium.org>
Reviewed-on: http://review.coreboot.org/2817
Tested-by: build bot (Jenkins)
Reviewed-by: Ronald G. Minnich <rminnich@gmail.com>
|
|
These events were initially for Chrome EC but they can be
applied to any EC.
Change-Id: I0eba9dbe8bde506e7f9ce18c7793399d40e6ab3b
Signed-off-by: Duncan Laurie <dlaurie@chromium.org>
Reviewed-on: http://review.coreboot.org/1746
Tested-by: build bot (Jenkins)
Reviewed-by: Ronald G. Minnich <rminnich@gmail.com>
|
|
We are seeing ME disabled and ME error events on some devices
and this extended info can help with debug.
Also fix a potential issue where if the log does manage to get
completely full it will never try to shrink it because the only
call to shrink the log happens after a successful event write.
Add a check at elog init time to shrink the log size.
Change-Id: Ib81dc231f6a004b341900374e6c07962cc292031
Signed-off-by: Duncan Laurie <dlaurie@chromium.org>
Reviewed-on: http://review.coreboot.org/1739
Tested-by: build bot (Jenkins)
Reviewed-by: Stefan Reinauer <stefan.reinauer@coreboot.org>
|
|
(elog portion, support in EC code pending)
- Use a new EC command to read the last post code
from the previous boot
- If the post code is not well-known final boot
or resume code then log it
Change-Id: Id6249e9a182243eb87c777edd56f48de72125e77
Signed-off-by: Duncan Laurie <dlaurie@chromium.org>
Reviewed-on: http://review.coreboot.org/1703
Reviewed-by: Marc Jones <marcj303@gmail.com>
Tested-by: build bot (Jenkins)
|
|
Recent changes in EC/Vboot/U-boot have completely broken
the logging of developer and recovery modes.
Recovery mode may not be in VBNV, so if that is zero and
yet we are in recovery mode then assume it is there because
the button/key was pressed.
Since there may not be any actual developer mode switch
we look if option rom is loaded and the system is not
in recovery mode and consider that as developer mode.
Change-Id: I70104877b24de477217e1ff5b3a019aef22343ec
Signed-off-by: Duncan Laurie <dlaurie@chromium.org>
Reviewed-on: http://review.coreboot.org/1346
Tested-by: build bot (Jenkins)
Reviewed-by: Ronald G. Minnich <rminnich@gmail.com>
|
|
This will log if the ME is disabled or has an error.
1) disable ME via EC console: gpioset PCH_HDA_SDO 1
2) boot the device
3) read eventlog with "mosys eventlog list"
71 | 2012-07-13 10:10:55 | Management Engine | Disabled
Change-Id: I9f6ee452d2aea76e6a5ea2cd50a50ff36245692a
Signed-off-by: Duncan Laurie <dlaurie@chromium.org>
Reviewed-on: http://review.coreboot.org/1345
Tested-by: build bot (Jenkins)
Reviewed-by: Ronald G. Minnich <rminnich@gmail.com>
|
|
The linux kernel contains an SMI driver that was written by
me (Duncan) and upstreamed a couple years ago called GSMI.
This driver will format a parameter buffer and pass pointers
to this parameter buffer to the SMI handler. It uses this to
generate events for kernel shutdown reasons: Clean, Panic, Oops,
etc.
This function expects to be passed pointers into the SMM state
save area that correspond to the prameter buffer and the return
code, which are typically EAX and EBX.
The format of the parameter buffer is defined in the kernel
driver so we implement the same interface here in order to be
compatible.
GSMI_CMD_HANDSHAKE: this is an early call that it does to try
and detect what kind of BIOS is running.
GSMI_CMD_SET_EVENT_LOG: this contains a parameter buffer that
has event type and data. The kernel-specific events are
translated here and raw events are passed through as well which
allows any run-time event to be added for testing.
GSMI_CMD_CLEAR_EVENT_LOG: this command clears the event log.
First the gsmi driver must be enabled in the kernel with
CONFIG_GOOGLE_GSMI and then events can be added via sysfs
and events are automatically generated for various kernel
shutdown reasons.
These can be seen in the event log as the 'Kernel Event' type:
169 | 2012-06-23 15:03:04 | Kernl Event | Clean Shutdown
181 | 2012-06-23 16:26:32 | Kernl Event | Oops
181 | 2012-06-23 16:26:32 | Kernl Event | Panic
Change-Id: Ic0a3916401f0d9811e4aa8b2c560657dccc920c1
Signed-off-by: Duncan Laurie <dlaurie@chromium.org>
Reviewed-on: http://review.coreboot.org/1316
Tested-by: build bot (Jenkins)
Reviewed-by: Ronald G. Minnich <rminnich@gmail.com>
|
|
This maintains a 32bit monotonically increasing boot counter
that is stored in CMOS and logged on every non-S3 boot when
the event log is initialized.
In CMOS the count is prefixed with a 16bit signature and
appended with a 16bit checksum.
This counter is incremented in sandybridge early_init which is
called by romstage. It is incremented early in order notice
when reboots happen after memory init.
The counter is then logged when ELOG is initialized and will
store the boot count as part of a 'System boot; event.
Reboot a few times and look for 'System boot' events in the
event log and check that they are increasing. Also verify
that the counter does NOT increase when resuming from S3.
171 | 2012-06-23 16:02:55 | System boot | 285
176 | 2012-06-23 16:26:00 | System boot | 286
182 | 2012-06-23 16:27:04 | System boot | 287
189 | 2012-06-23 16:31:10 | System boot | 288
Change-Id: I23faeafcf155edfd10aa6882598b3883575f8a33
Signed-off-by: Duncan Laurie <dlaurie@chromium.org>
Reviewed-on: http://review.coreboot.org/1315
Tested-by: build bot (Jenkins)
Reviewed-by: Ronald G. Minnich <rminnich@gmail.com>
|
|
This standared SMBIOS 0able describes the location and format
of the event log to the OS and applications. In this case the
pointer is a 32bit physical address pointer to the log in
memory mapped flash.
Look for SMBIOS type15 entry with 'dmidecode -t 15'
Handle 0x0004, DMI type 15, 23 bytes
System Event Log
Area Length: 4095 bytes
Header Start Offset: 0x0000
Header Length: 8 bytes
Data Start Offset: 0x0008
Access Method: Memory-mapped physical 32-bit address
Access Address: 0xFFB6F000
Status: Valid, Not Full
Change Token: 0x00000000
Header Format: OEM-specific
Supported Log Type Descriptors: 0
Change-Id: I1e7729e604000f197e26e69991a2867e869197a6
Signed-off-by: Duncan Laurie <dlaurie@chromium.org>
Reviewed-on: http://review.coreboot.org/1314
Tested-by: build bot (Jenkins)
Reviewed-by: Ronald G. Minnich <rminnich@gmail.com>
|
|
This is based around the SMBIOS event log specification but
expanded with OEM event types to support more specific and
relevant system events.
It requires flash storage and a minimum 4K block (or flash block
size) that should be allocated in the FMAP.
A copy of the event log is maintained in memory for convenience
and speed and the in-memory copy is written to flash at specific
points.
The log is automatically shunk when it reaches a configurable
full threshold in order to not get stuck with a full log that
needs OS help to clear.
ELOG implements the specification published here:
http://code.google.com/p/firmware-event-log/wiki/FirmwareEventLogDesign
And is similar to what we use in other firmware at Google.
This implementation does not support double-buffered flash
regions. This is done because speed is valued over the log
reliability and it keeps the code simpler for the first version.
This is a large commit and by itself it just provides a new
driver that is made available to coreboot. Without additional
patches it is not very useful, but the end result is an event
log that will contain entries like this:
171 | 2012-06-23 16:02:55 | System boot | 285
172 | 2012-06-23 16:02:55 | EC Event | Power Button
173 | 2012-06-23 16:02:55 | SUS Power Fail
174 | 2012-06-23 16:02:55 | System Reset
175 | 2012-06-23 16:02:55 | ACPI Wake | S5
Change-Id: I985524c67f525c8a268eccbd856c1a4c2a426889
Signed-off-by: Duncan Laurie <dlaurie@chromium.org>
Reviewed-on: http://review.coreboot.org/1311
Tested-by: build bot (Jenkins)
Reviewed-by: Ronald G. Minnich <rminnich@gmail.com>
|