| mfsmaster.cfg(5) | This is part of MooseFS | mfsmaster.cfg(5) |
mfsmaster.cfg - main configuration file for mfsmaster
The file mfsmaster.cfg contains configuration of MooseFS master process.
Syntax is:
Lines starting with # character are ignored as comments.
Changes in this section require process restart.
Changes in this section require only process reload.
Changes in this section require only process reload.
Changes in this section require only process reload.
Changes in this section require only process reload.
Changes in this section require only process reload.
Changes in this section require only process reload.
Changes in this section require only process reload.
Changes in this section require only process reload.
For config variables that define time without requiring a single, specific unit, time can be defined as a number of seconds (integer) or a time period in one of two possible formats:
first format: #.#T where T is one of: s-seconds, m-minutes, h-hours, d-days or w-weeks; fractions of seconds will be rounded to full seconds
second format: #w#d#h#m#s, any number of definitions can be ommited, but the remaining definitions must be in order (so #d#m is still a valid definition, but #m#d is not); ranges: s,m: 0 to 59, h: 0 to 23, d: 0 t o 6, w is unlimited and the first definition is also always unlimited (i.e. for #d#h#m d will be unlimited)
Examples:
1.5h is the same as 1h30m, is the same as 90m, is the same as 5400s, is the same as 5400
2.5d is the same as 2d12h, is the same as 60h; 1d36h is not a valid time period (h is not the first definition, so it is bound by range 0 to 23)
1.03m is the same as 62s (61.8 seconds will be rounded up to 62)
Chunks in master are tested in a loop. Speed (or frequency) is regulated by two options: CHUNKS_LOOP_MIN_TIME and CHUNKS_LOOP_MAX_CPS. First defines minimal time between iterations of the loop and second defines maximal number of chunk tests per second. Typically at the beginning, when the number of chunks is small, time is constant, regulated by CHUNK_LOOP_MIN_TIME, but the when number of chunks becomes bigger, then time of loop can increase according to CHUNKS_LOOP_MAX_CPS.
Example: CHUNKS_LOOP_MIN_TIME is set to 300, CHUNKS_LOOP_MAX_CPS is set to 100000 and there is 1000000 (one million) chunks in the system. 1000000/100000 = 10, which is less than 300, so one loop iteration will take 300 seconds. With 1000000000 (one billion) chunks the system needs 10000 seconds for one iteration of the loop.
Deletion limits are defined as 'soft' and 'hard' limit. When number of chunks to delete increases from loop to loop, current limit can be temporarily increased above soft limit, but never above hard limit.
Replication limits are divided into five cases:
Usually first number should be greater than or equal to second, second greater than or equal to third, and fourth greater than or equal to third ( 1st >= 2nd >= 3rd <= 4th ). Fifth limit should be equal or greater than any of the other limits. If one number is given, then all limits are set to this number (for backward compatibility). If only four numbers are given, the fifth will be set as maximum of the four (also for backward compatibility).
Whenever chunkserver load is higher than CS_HEAVY_LOAD_THRESHOLD and CS_HEAVY_LOAD_RATIO_THRESHOLD times higher than average load, then chunkserver is switched into 'grace' mode. Chunkserver stays in grace mode for CS_HEAVY_LOAD_GRACE_PERIOD seconds.
There are five possible values for ATIME_MODE (all other values are treated as 0):
You can reserve space for superuser using RESERVE_SPACE option. You can define it as number of bytes, percent of total space, capacity of biggest chunkserver, etc.
When your network has two (or more) IP classes you may want to use one network for standard communication between MFS modules and separate network only for I/O. It can be done by setting REMAP_BITS, REMAP_SOURCE_IP_CLASS and REMAP_DESTINATION_IP_CLASS. When you set these options then master will change internally IP addresses of chunkservers and will send them as chunk locations, so clients will make connections with chunkservers using new (destination) IP for all I/O, but still communicate with master using original (source) IP. Also chunkservers will use original IP to communicate with master, but they will use new IP's to communicate between themselves during replication. Beware that all clients and chunkservers must have access to both networks, but masters, metaloggers etc. will need only access to the source network.
When your clients are separated into two or more LAN or VLAN networks, you may want them to connect to masters and chunkservers using IPs from their network. It can be done by setting MULTILAN_BITS and MULTILAN_CLASSES. Each time a client connects, the master will check whether the connection came from one of the defined LAN classes and if yes, it will remap the first MULTILAN_BITS of any chunkserver IP before it sends the chunkserver's IP to the client. Connections coming from other IP addresses will be treated as usual (i.e. original chunkserver IPs will be sent in response). All masters and chunkservers need to have one IP from each of the defined MULTILAN_CLASSES and one module (chunkserver or master) needs to have the same IP suffix in each class. Proper DNS configuration is also required: clients in each LAN must either get different IP when querying DNS about master host or must use different master host names that are resolved to IPs in their class. See mfsmount(8), mfsbdev(8) for more info about master host name.
The two above sets of options can be used together. One purpose would be to create a separate network for replication of data between chunkservers while also maintaning several separate networks (LANs) for separate sets of clients. For example, with the following configuration:
REMAP_BITS = 24
REMAP_SOURCE_IP_CLASS = 10.0.0.0
REMAP_DESTINATION_IP_CLASS = 10.0.1.0
MULTILAN_BITS = 24
MULTILAN_CLASSES = 192.168.1.0, 192.168.2.0, 192.168.3.0, 10.0.1.0
all network traffic from clients matching one of the LAN classes will be handled on that network (so a client with IP 192.168.1.17 will connect to the master and chunkservers using IPs with prefix 192.168.1 and a client with IP 192.168.2.13 will connect to the master and chunkservers using IPs with prefix 192.168.2), all metadata traffic between master and chunkservers will be handled on network with prefix 10.0.0 and all direct communication between chunkservers (i.e. chunk replications) will be handled on network with prefix 10.0.1. Be aware that any client from outside of the defined LAN classes will connect to the chunkservers via IPs defined by REMAP_DESTINATION_IP_CLASS. This also assumes proper DNS configuration, that is, if master server uses IP suffix 1, a client with IP 192.168.1.17 should resolve master host name as 192.168.1.1 and a client with IP 192.168.2.13 should resolve master host name as 192.168.2.1. A client from outside of the defined LAN classes may use any of master server IPs, although preferably 10.0.0.1
Masters save metadata to a file on a local disk. The exact times of these operations are regulated by four variables: METADATA_SAVE_OFFSET, METADATA_SAVE_FREQ, (pro only) METADATA_DOWNLOAD_FREQ, (pro only) METADATA_CRCCHECK_FREQ.
A single master will save metadata every METADATA_SAVE_FREQ hours. First save of the day happens at midnight, every other one is after METADATA_SAVE_FREQ from previous one. This can be changed with METADATA_SAVE_OFFSET. METADATA_SAVE_OFFSET set to a single value will mean minutes. So settings like this:
METADATA_SAVE_FREQ = 2
METADATA_SAVE_OFFSET = 15
will mean saving at 00:15, 02:15, 04:15 etc., up to 22:15. If hour is provided, this is also taken into account, but only as modulo, so:
METADATA_SAVE_FREQ = 2
METADATA_SAVE_OFFSET = 1:15
will behave exactly the same as:
METADATA_SAVE_FREQ = 2
METADATA_SAVE_OFFSET = 5:15
and will save metadata at 01:15, 03:15, 05:15 etc.
All times mentioned above are calculated in UTC, unless "L" suffix is used. So, if your servers are in CET zone (UTC-1), this setting:
METADATA_SAVE_FREQ = 2
METADATA_SAVE_OFFSET = 1:15
will mean the master will save metadata at 01:15, 3:15, etc. UTC, which means 0:15, 2:15, etc. local (CET) time. To save at 01:15 (and every 2 hours from that) local time, you need to write:
METADATA_SAVE_FREQ = 2
METADATA_SAVE_OFFSET = 1:15L
(pro only) With multiple masters the leader will save metadata according to the setting from METADATA_CRCCHECK_FREQ, but also taking into account the offset. So setting like:
METADATA_CRCCHECK_FREQ = 12
METADATA_SAVE_FREQ = 2
METADATA_SAVE_OFFSET = 1:15
will mean that the leader saves (and compares against followers) metadata at 01:15 and 13:15, while followers save at 01:15, 03:15, 05:15 etc.
METADATA_DOWNLOAD_FREQ means the leader will download metadata from a follower and that will also take into account the METADATA_SAVE_OFFSET in the manner identical that METADATA_CRCCHECK_FREQ does.
Times mentioned above are calculated in UTC. So, if your servers are in PST zone (UTC-8), this setting:
METADATA_CRCCHECK_FREQ = 12
METADATA_SAVE_FREQ = 2
METADATA_SAVE_OFFSET = 1:15
will mean the leader will save metadata at 01:15 and 13:15 UTC, which means 17:15 and 5:15 (5:15 PM and 5:15 AM) local (PST) time. To save at 01:15 and 13:15 (1:15 AM and 1:15 PM) local time, you need to write:
METADATA_CRCCHECK_FREQ = 12
METADATA_SAVE_FREQ = 2
METADATA_SAVE_OFFSET = 1:15L
Setting:
METADATA_SAVE_OFFSET = 0L
can be used to adjust saving times to local timezone without changing the default metadata saving schedule.
Copyright (C) 2024 Jakub Kruszona-Zawadzki, Saglabs SA
This file is part of MooseFS.
MooseFS is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 2 (only).
MooseFS is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with MooseFS; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02111-1301, USA or visit http://www.gnu.org/licenses/gpl-2.0.html
mfsmaster(8), mfsexports.cfg(5) mfstopology.cfg(5) mfsipmap.cfg(5)
| September 2024 | MooseFS 4.56.6-1 |