Martin Mucha has uploaded a new change for review.
Change subject: core: MacPoolManager strategy, alternative implementation, test
for benchmarks
......................................................................
core: MacPoolManager strategy, alternative implementation, test for
benchmarks
* internals of MacPoolManager moved to separate implementation of
MacPoolManagerStrategy (MacPoolManagerStrategyOriginal) to allow
more implementations and comparing between them.
* introduced new alternative implentation of MacPoolManager
(MacPoolManagerStragetyRanges)
which is less eager browsing its internal structures and
less in need of RAM. There is still need of counting how many
times is each MAC used; counts of macs from each user specified
mac range are stored in separate int array(instead of hashmap).
Counts of user specified MAC outside from that MAC range are
still stored in HashMap, but modifiable int wrappers are used
instead of java.lang.Integer to cut down calls to HashMap.
Acquiring last unused MAC from MAC range would cost O(n) to find
MAC which is used zero times, which is too much, so BitSet is used
to flag each index in forementioned int array as used or not used.
*user can input multiple MAC ranges and original implementation
dealt with duplicates using HashSet, which means potentially lot
of unnecessary effort and copying data in RAM from one place to
another. New class RangesWithoutOverlaps was introduced, which
takes user input and transfors it into N intervals
(as few as possible) without overlaps.
*MacPoolManager is singleton; only 'getInstance()' has to be
static other methods should be instance methods. Having them
static is mixing concepts.
---
space costs(for 1 000 000 MACs range):
(these measurements need not be accurate, but it's probably the
best you can get from JVM):
Measurement for: MacPoolManagerOriginal------
Time: 794 ms
size: 120389288 bytes (114 MBytes)
object count: 3000018
=============================================
Measurement for: MacPoolManagerRanges--------
Time: 19 ms
size: 4125568 bytes (3 MBytes)
object count: 20
=============================================
time costs(for 1 000 000 MACs range):
(I've wrote few test trying to verify that both implementation
does the same thing, measuring how long it takes to react to
exactly same 'client' request. These measurements need not be
accurate, but it's probably the best you can get from JVM):
addRemoveComparison:
(init 1M MAC range, acquire half of them, release half of them)
originalAdd: 674ms
originalRm: 375ms
rangesAdd: 506ms
rangesRm: 145ms
fullFillTest
(allocate every MAC in given range, each of them is named by
'client' -- allocate *this* mac, allocate ...)
originalAdd: 1123ms
rangesAdd: 0913ms
fullFillTest2
(allocate every MAC in given range without naming it -- allocate
*some* mac)
originalAdd: 440765ms
rangesAdd: 009305ms
allocatingOnTheEndOfArrayPerformance
(try to remove 1000 mac from the end of the range and then
allocated them back(without specifying which ones were deleted))
originalAdd: 1235ms
originalRm: 0000ms
rangesAdd: 0014ms
rangesRm: 0000ms
Change-Id: I69a5c1b3b43966e49fa6039597c06966ce514618
Bug-Url: https://bugzilla.redhat.com/1063064
Signed-off-by: Martin Mucha <mmu...@redhat.com>
---
M backend/manager/modules/bll/pom.xml
M
backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/network/MacPoolManager.java
A
backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/network/MacPoolManagerRanges.java
A
backend/manager/modules/bll/src/test/java/org/ovirt/engine/core/bll/network/MacPoolManagerInitializationTimeTest.java
A
backend/manager/modules/bll/src/test/java/org/ovirt/engine/core/bll/network/MacPoolManagerTestUtils.java
A
backend/manager/modules/bll/src/test/java/org/ovirt/engine/core/bll/network/MacPoolManagerValidityTest.java
M
backend/manager/modules/bll/src/test/java/org/ovirt/engine/core/bll/network/VmInterfaceManagerTest.java
7 files changed, 1,030 insertions(+), 2 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/ovirt-engine refs/changes/05/26405/1
diff --git a/backend/manager/modules/bll/pom.xml
b/backend/manager/modules/bll/pom.xml
index ed7a977..99681e7 100644
--- a/backend/manager/modules/bll/pom.xml
+++ b/backend/manager/modules/bll/pom.xml
@@ -19,6 +19,14 @@
</properties>
<dependencies>
+
+ <dependency>
+ <groupId>com.github.stephenc</groupId>
+ <artifactId>jamm</artifactId>
+ <version>0.2.5</version>
+ <scope>test</scope>
+ </dependency>
+
<dependency>
<groupId>${engine.groupId}</groupId>
<artifactId>compat</artifactId>
diff --git
a/backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/network/MacPoolManager.java
b/backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/network/MacPoolManager.java
index 0abd3c8..0bcb9d2 100644
---
a/backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/network/MacPoolManager.java
+++
b/backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/network/MacPoolManager.java
@@ -15,7 +15,7 @@
final Integer maxMacsCountInPool =
Config.getValue(ConfigValues.MaxMacsCountInPool);
final String macPoolRanges =
Config.getValue(ConfigValues.MacPoolRanges);
final Boolean allowDuplicates =
Config.getValue(ConfigValues.AllowDuplicateMacAddresses);
- macPoolManagerStrategy = new
MacPoolManagerOriginal(maxMacsCountInPool, macPoolRanges, allowDuplicates);
+ macPoolManagerStrategy = new MacPoolManagerRanges(maxMacsCountInPool,
macPoolRanges, allowDuplicates);
}
public static MacPoolManager getInstance() {
diff --git
a/backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/network/MacPoolManagerRanges.java
b/backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/network/MacPoolManagerRanges.java
new file mode 100644
index 0000000..3e98841
--- /dev/null
+++
b/backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/network/MacPoolManagerRanges.java
@@ -0,0 +1,503 @@
+package org.ovirt.engine.core.bll.network;
+
+import java.util.Arrays;
+import java.util.BitSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+import org.ovirt.engine.core.common.AuditLogType;
+import org.ovirt.engine.core.common.businessentities.network.VmNic;
+import org.ovirt.engine.core.common.errors.VdcBLLException;
+import org.ovirt.engine.core.common.errors.VdcBllErrors;
+import org.ovirt.engine.core.common.utils.Pair;
+import org.ovirt.engine.core.dal.dbbroker.DbFacade;
+import org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector;
+import org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogableBase;
+import org.ovirt.engine.core.utils.MacAddressRangeUtils;
+import org.ovirt.engine.core.utils.log.Log;
+import org.ovirt.engine.core.utils.log.LogFactory;
+
+public class MacPoolManagerRanges implements MacPoolManagerStrategy{
+
+ private static final Log log =
LogFactory.getLog(MacPoolManagerRanges.class);
+
+ private final int maxMacsInPool;
+ private final String rangesString;
+
+ private final ReentrantReadWriteLock lockObj = new
ReentrantReadWriteLock();
+ private final Boolean allowDuplicates;
+ private boolean initialized;
+ private MacsStorage macsStorage;
+
+ public MacPoolManagerRanges(Integer maxMacsInPool, String rangesString,
Boolean allowDuplicates) {
+ this.maxMacsInPool = maxMacsInPool;
+ this.rangesString = rangesString;
+ this.allowDuplicates = allowDuplicates;
+ }
+
+ @Override
+ public void initialize() {
+ lockObj.writeLock().lock();
+ try {
+ log.infoFormat("Start initializing "+getClass().getSimpleName());
+
+ this.macsStorage = initRanges(rangesString);
+
+
+ List<VmNic> interfaces = getVmNicInterfacesFromDB();
+
+ for (VmNic iface : interfaces) {
+ forceAddMac(iface.getMacAddress());
+ }
+ initialized = true;
+ log.infoFormat("Finished initializing. Available MACs in pool:
{0}", this.macsStorage.getAvailableMacsCount());
+ } catch (Exception ex) {
+ log.errorFormat("Error in initializing MAC Addresses pool
manager.", ex);
+ } finally {
+ lockObj.writeLock().unlock();
+ }
+ }
+
+ MacsStorage initRanges(String rangesString) {
+ List<Pair<Long, Long>> rangesBoundaries =
MacAddressRangeUtils.rangesStringToRangeBoundaries(rangesString);
+ int macsToCreate = maxMacsInPool;
+ MacsStorage macsStorage = new MacsStorage(this.allowDuplicates);
+ for (Pair<Long, Long> rangesBoundary : rangesBoundaries) {
+ macsToCreate -= macsStorage.addRange(rangesBoundary.getFirst(),
rangesBoundary.getSecond(), macsToCreate);
+ }
+
+ if (macsStorage.noAvailableMacs()) {
+ throw new
VdcBLLException(VdcBllErrors.MAC_POOL_INITIALIZATION_FAILED);
+ } else {
+ return macsStorage;
+ }
+ }
+
+ //to allow testing; may be fixed after DI is used.
+ List<VmNic> getVmNicInterfacesFromDB() {
+ return DbFacade.getInstance().getVmNicDao().getAll();
+ }
+
+ //to allow testing; may be fixed after DI is used.
+ void logMacPoolEmpty() {
+ AuditLogableBase logable = new AuditLogableBase();
+ AuditLogDirector.log(logable, AuditLogType.MAC_POOL_EMPTY);
+ }
+
+ @Override
+ public String allocateNewMac() {
+ lockObj.writeLock().lock();
+ try {
+ return allocateNewMacImpl(1)[0];
+ } finally {
+ lockObj.writeLock().unlock();
+ }
+ }
+
+ private String[] allocateNewMacImpl(int numberOfMacs) {
+ if (!initialized) {
+ logInitializationError("Failed to allocate new Mac address.");
+ throw new VdcBLLException(VdcBllErrors.MAC_POOL_NOT_INITIALIZED);
+ }
+ if (macsStorage.getAvailableMacsCount() < numberOfMacs) { //TODO
MM: allocate just those remaining or none?
+ throw new VdcBLLException(VdcBllErrors.MAC_POOL_NO_MACS_LEFT);
+ }
+
+ long[] macs = macsStorage.allocateAvailableMac(numberOfMacs);
+ Arrays.sort(macs);
+ if (macsStorage.noAvailableMacs()) {
+ logMacPoolEmpty();
+ }
+
+ return MacAddressRangeUtils.macAddressIntToString(macs);
+ }
+
+ @Override
+ public int getAvailableMacsCount() {
+ lockObj.readLock().lock();
+ try {
+ if (!initialized) {
+ logInitializationError("Failed to get available Macs count.");
+ throw new
VdcBLLException(VdcBllErrors.MAC_POOL_NOT_INITIALIZED);
+ }
+
+ int availableMacsSize = macsStorage.getAvailableMacsCount();
+ log.debugFormat("Number of available Mac addresses = {1}",
availableMacsSize);
+ return availableMacsSize;
+ } finally {
+ lockObj.readLock().unlock();
+ }
+ }
+
+ @Override
+ public void freeMac(String mac) {
+ lockObj.writeLock().lock();
+ try {
+ if (!initialized) {
+ logInitializationError("Failed to free mac address " + mac + "
.");
+ } else {
+ macsStorage.freeMac(MacAddressRangeUtils.macStringToLong(mac));
+ }
+ } finally {
+ lockObj.writeLock().unlock();
+ }
+ }
+
+ private void logInitializationError(String message) {
+ log.errorFormat("The MAC addresses pool is not initialized");
+ AuditLogableBase loggable = new AuditLogableBase();
+ loggable.addCustomValue("Message", message);
+ AuditLogDirector.log(loggable,
AuditLogType.MAC_ADDRESSES_POOL_NOT_INITIALIZED);
+ }
+
+ /**
+ * Add given MAC address if possible.
+ * Add user define mac address Function return false if the mac is in use
+ * @return true if MAC was added successfully, and false if the MAC is in
use and
+ * {@link
org.ovirt.engine.core.common.config.ConfigValues#AllowDuplicateMacAddresses} is
set to false
+ */
+ @Override
+ public boolean addMac(String mac) {
+ lockObj.writeLock().lock();
+ try {
+ boolean added =
this.macsStorage.useMac(MacAddressRangeUtils.macStringToLong(mac));
+ if (macsStorage.noAvailableMacs()) {
+ logMacPoolEmpty();
+ }
+ return added;
+
+
+ } finally {
+ lockObj.writeLock().unlock();
+ }
+ }
+
+ /**
+ * Add given MAC address, regardless of it being in use.
+ *
+ */
+ @Override
+ public void forceAddMac(String mac) {
+ lockObj.writeLock().lock();
+ try {
+
this.macsStorage.useMacNoDuplicityCheck(MacAddressRangeUtils.macStringToLong(mac));
+ if (macsStorage.noAvailableMacs()) {
+ logMacPoolEmpty();
+ }
+ } finally {
+ lockObj.writeLock().unlock();
+ }
+ }
+
+ @Override
+ public boolean isMacInUse(String mac) {
+ lockObj.readLock().lock();
+ try {
+ return
this.macsStorage.isMacInUse(MacAddressRangeUtils.macStringToLong(mac));
+
+ } finally {
+ lockObj.readLock().unlock();
+ }
+ }
+
+ @Override
+ public void freeMacs(List<String> macs) { //TODO MM: how about some
duplicities here? Release it twice or what?
+ if (!macs.isEmpty()) {
+ lockObj.writeLock().lock();
+ try {
+ if (!initialized) {
+ logInitializationError("Failed to free MAC addresses.");
+ }
+ for (String mac : macs) {
+
macsStorage.freeMac(MacAddressRangeUtils.macStringToLong(mac));
+ }
+
+ } finally {
+ lockObj.writeLock().unlock();
+ }
+ }
+ }
+
+ /**
+ * Allocates mac addresses according to the input, sorting them in
ascending order
+ *
+ * @param numberOfAddresses
+ * The number of MAC addresses to allocate
+ * @return The list of MAC addresses, sorted in ascending order
+ */
+ @Override
+ public List<String> allocateMacAddresses(int numberOfAddresses) {
+ lockObj.writeLock().lock();
+ try {
+ String[] macAddresses = this.allocateNewMacImpl(numberOfAddresses);
+ return Arrays.asList(macAddresses);
+ } finally {
+ lockObj.writeLock().unlock();
+ }
+ }
+
+ private static class MacsStorage {
+ private final Boolean allowDuplicates;
+ private List<Range> ranges = new LinkedList<>();
+ private ObjectCounter<Long> customMacs;
+
+ public MacsStorage(Boolean allowDuplicates) {
+ this.allowDuplicates = allowDuplicates;
+ customMacs = new ObjectCounter<>(this.allowDuplicates);
+ }
+
+ public long addRange(long first, long second, int maxMacsInPool) {
+
+ long i = first;
+ int numberOfNonMultiCasts = 0;
+
+ for (; i < second && numberOfNonMultiCasts < maxMacsInPool; i++) {
+ if ((i & MacAddressRangeUtils.MAC_ADDRESS_MULTICAST_BIT) == 0)
{ //not a multicast
+ numberOfNonMultiCasts++;
+ }
+ }
+
+ long size = i - first;
+ if (size > Integer.MAX_VALUE) {
+ throw new IllegalArgumentException("Not supported that wide
ranges(" + size + ").");
+ }
+
+ if (numberOfNonMultiCasts > 0) {
+ ranges.add(new Range(first, (int)size,
numberOfNonMultiCasts)); //java. arrays/collections can not be bigger than
2^32.
+ }
+ return numberOfNonMultiCasts;
+ }
+
+ public boolean useMac(long mac) {
+ return useMac(mac, allowDuplicates);
+ }
+
+ private boolean useMac(long mac, Boolean allowDuplicates) {
+ testForMultiCastMac(mac);
+
+ Range range = getRange(mac);
+ if (range == null) {
+ return this.customMacs.add(mac);
+ } else {
+ return range.use(mac, allowDuplicates);
+ }
+ }
+
+ public void useMacNoDuplicityCheck(long mac) {
+ useMac(mac, true);
+ }
+
+ public boolean isMacInUse(long mac) {
+ testForMultiCastMac(mac);
+
+ Range range = getRange(mac);
+ return range != null ? range.isAllocated(mac) :
this.customMacs.contains(mac);
+ }
+
+ public boolean freeMac(long mac) {
+ testForMultiCastMac(mac);
+
+ Range range = getRange(mac);
+ if (range != null) {
+ return range.freeMac(mac);
+ } else {
+ return this.customMacs.remove(mac);
+ }
+ }
+
+ public boolean noAvailableMacs() {
+ for (Range range : ranges) {
+ if (range.getNotUsedCount() > 0) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ public long[] allocateAvailableMac(int numberOfMacs) {
+ return getRangeWithAvailableMac().allocateMac(numberOfMacs);
+ }
+
+ private Range getRangeWithAvailableMac() {
+ for (Range range : ranges) {
+ if (range.getNotUsedCount() > 0) {
+ return range;
+ }
+ }
+ return null;
+ }
+
+ public int getAvailableMacsCount() {
+ int count = 0;
+ for (Range range : ranges) {
+ count+=range.getNotUsedCount();
+ }
+ return count;
+ }
+
+ private void testForMultiCastMac(long mac) {
+ if (MacAddressRangeUtils.macHasMultiCastBitSet(mac)) {
+ throw new IllegalArgumentException("You're trying to work with
multicast address."); //TODO MM: should this be prohibited?
+ }
+ }
+
+ private Range getRange(long mac) {
+ for (Range range : ranges) {
+ if (range.offset <= mac && (range.offset + range.size) >= mac)
{
+ return range;
+ }
+ }
+ return null;
+ }
+
+ private static class Range {
+ private final long offset;
+ private final int size;
+ private final int[] macUsageCount;
+ private int _notUsedCount;
+
+ private NotUsedMacs notUsedMacs;
+
+ public interface NotUsedMacs {
+ void removeMacFromNotUsed(int macIndex);
+
+ void addMacAmongFree(int macIndex);
+
+ long unusedMac(long offset);
+ }
+
+ private static class NotUsedMacsBitSet implements NotUsedMacs {
+
+ private BitSet bitSet;
+
+ public NotUsedMacsBitSet(int size) {
+ this.bitSet = new BitSet(size);
+ bitSet.set(0, size, true);
+ }
+
+ @Override
+ public void removeMacFromNotUsed(int macIndex) {
+ this.bitSet.set(macIndex, false);
+ }
+
+ @Override
+ public void addMacAmongFree(int macIndex) {
+ this.bitSet.set(macIndex, true);
+ }
+
+ @Override
+ public long unusedMac(long offset) {
+ int start = 0;
+ int notUsedMacIndex = 0;
+ while (notUsedMacIndex != -1) {
+ notUsedMacIndex = this.bitSet.nextSetBit(start);
+ start = notUsedMacIndex;
+ long mac = offset + notUsedMacIndex;
+ if (MacAddressRangeUtils.macHasMultiCastBitSet(mac)) {
+ removeMacFromNotUsed(notUsedMacIndex);
+ } else {
+ return mac;
+ }
+ }
+
+ throw new IllegalStateException("There should be available
mac, but there isn't any");
+ }
+ }
+
+ public Range(long offset, int size, int numberOfNonMultiCasts) {
+ this.offset = offset;
+ this.size = size;
+ this._notUsedCount = numberOfNonMultiCasts;
+
+ macUsageCount = new int[size];
+ this.notUsedMacs = new NotUsedMacsBitSet(size);
+ }
+
+ /**
+ *
+ * @param mac mac to add
+ * @return if mac was used (it's usage count was increased). I.e.
if it was not used, it's used now, or
+ * it was used and duplicates are allowed so it's not used some
more.
+ */
+ public boolean use(long mac, boolean allowDuplicates) {
+ try {
+ int index = macToArrayIndex(mac);
+ int currentCount = test(index);
+
+ if (currentCount == 0) {
+ _notUsedCount --;
+
this.notUsedMacs.removeMacFromNotUsed(macToArrayIndex(mac));
+ macUsageCount[index]+=1;
+ return true;
+ }
+
+ if (allowDuplicates) {
+ macUsageCount[index]+=1;
+ return true;
+ }
+
+ return false;
+ } catch (Exception e) {
+ throw new RuntimeException("failed use during adding mac:
" + mac + ", index = " + (mac - offset), e);
+ }
+
+ }
+
+ int test(int index) {
+ return macUsageCount[index];
+ }
+
+ int macToArrayIndex(long mac) {
+ return (int) (mac - offset);
+ }
+
+ public boolean isAllocated(long mac) {
+ int index = macToArrayIndex(mac);
+ int currentCount = macUsageCount[index];
+ return currentCount > 0;
+ }
+
+ public boolean freeMac(long mac) {
+ int index = macToArrayIndex(mac);
+ int currentCount = macUsageCount[index];
+ if (currentCount == 0) {
+ return false;
+ } else {
+ macUsageCount[index] -= 1;
+
+ if (currentCount == 1) { //was there and now
it's gone. It's free.\
+ this.notUsedMacs.addMacAmongFree(macToArrayIndex(mac));
+ _notUsedCount++;
+ return true;
+ } else {
+ return false;
+ }
+
+ }
+ }
+
+ public int getNotUsedCount() {
+ return _notUsedCount;
+ }
+
+ public long[] allocateMac(int numberOfMacs) {
+ long[] result = new long[numberOfMacs];
+
+ int count = 0;
+ while (count < numberOfMacs) {
+ final long mac = notUsedMacs.unusedMac(offset);
+ use(mac, false); //well duplicates may be allowed,
but we're using unallocated mac.
+ result[count++] = mac;
+ }
+
+ return result;
+ }
+
+ }
+
+
+ }
+
+
+}
diff --git
a/backend/manager/modules/bll/src/test/java/org/ovirt/engine/core/bll/network/MacPoolManagerInitializationTimeTest.java
b/backend/manager/modules/bll/src/test/java/org/ovirt/engine/core/bll/network/MacPoolManagerInitializationTimeTest.java
new file mode 100644
index 0000000..055eed7
--- /dev/null
+++
b/backend/manager/modules/bll/src/test/java/org/ovirt/engine/core/bll/network/MacPoolManagerInitializationTimeTest.java
@@ -0,0 +1,35 @@
+package org.ovirt.engine.core.bll.network;
+
+import org.junit.Ignore;
+import org.junit.Test;
+
+//add cmd-line argument before run: "
-javaagent:${HOME}/.m2/repository/com/github/stephenc/jamm/0.2.5/jamm-0.2.5.jar
"
+
+@Ignore("Maven is not configured to run this automatically, you have to run it
manually.")
+public class MacPoolManagerInitializationTimeTest {
+
+ public static final int MAX_MACS_IN_POOL = 1000000;
+// public static final int MAX_MACS_IN_POOL = 100000;
+// public static final int MAX_MACS_IN_POOL = 1000;
+
+ @Test
+ public void testOriginal() throws Exception {
+
measureTimeAndMemoryConsumption(MacPoolManagerOriginal.class.getSimpleName(),
+
MacPoolManagerTestUtils.createMacPoolManagerOriginalSpy(MAX_MACS_IN_POOL,
false));
+
+
measureTimeAndMemoryConsumption(MacPoolManagerRanges.class.getSimpleName(),
+
MacPoolManagerTestUtils.createMacPoolManagerRangesSpy(MAX_MACS_IN_POOL, false));
+
+ }
+
+ void measureTimeAndMemoryConsumption(final String strategyName, final
MacPoolManagerStrategy strategy) {
+ new MacPoolManagerTestUtils.Measurement(strategyName) {
+ @Override
+ protected Object runAndGetObjectToMeasure() {
+ strategy.initialize();
+ return strategy;
+ }
+ }.measure();
+ }
+
+}
diff --git
a/backend/manager/modules/bll/src/test/java/org/ovirt/engine/core/bll/network/MacPoolManagerTestUtils.java
b/backend/manager/modules/bll/src/test/java/org/ovirt/engine/core/bll/network/MacPoolManagerTestUtils.java
new file mode 100644
index 0000000..6c256c7
--- /dev/null
+++
b/backend/manager/modules/bll/src/test/java/org/ovirt/engine/core/bll/network/MacPoolManagerTestUtils.java
@@ -0,0 +1,115 @@
+package org.ovirt.engine.core.bll.network;
+
+import java.lang.management.ManagementFactory;
+import java.lang.management.ThreadMXBean;
+import java.util.Collections;
+import java.util.List;
+
+import org.github.jamm.MemoryMeter;
+import org.mockito.Mockito;
+import org.ovirt.engine.core.common.businessentities.network.VmNic;
+
+class MacPoolManagerTestUtils {
+
+ public static final String RANGES_FROM = "00:1A:4A:01:00:00";
+ public static final String RANGES_TO = "00:1A:4A:FF:FF:FF";
+ public static final String RANGES_STRING = RANGES_FROM + "-" + RANGES_TO;
+ public static final boolean ALLOW_DUPLICATES = false;
+ public static final long NANOS_IN_MS = 1000 * 1000;
+ public static final long BYTES_IN_MB = 1024 * 1024;
+ private static final ThreadMXBean bean =
ManagementFactory.getThreadMXBean();
+
+ private MacPoolManagerTestUtils() {
+ }
+
+ public static long timeStamp() {
+ return bean.isCurrentThreadCpuTimeSupported() ?
bean.getCurrentThreadCpuTime() : 0L;
+
+ }
+
+
+ public static abstract class Measurement {
+ private String desc;
+
+ protected Measurement(String desc) {
+ this.desc = desc;
+ }
+
+ public void prepare() {
+ System.out.println("Preparing ...");
+ long start = timeStamp();
+ runAndGetObjectToMeasure();
+ long end = timeStamp();
+ System.out.printf("\tPreparation Time: %d ms\n", ((end - start) /
NANOS_IN_MS));
+
+ System.out.println("calculating ...");
+
+ }
+
+ public void measure() {
+ prepare();
+ System.out.println("-----");
+ MemoryMeter meter = new MemoryMeter();
+ long start = timeStamp();
+ Object measureThis = runAndGetObjectToMeasure();
+ long end = timeStamp();
+ System.out.println("Measurement for: " + desc +
"--------------------");
+ System.out.printf("\tTime: %d ms\n", ((end - start) /
NANOS_IN_MS));
+ long size = meter.measureDeep(measureThis);
+ System.out.printf("\tsize: %d bytes (%d MBytes)\n", size, (size /
BYTES_IN_MB));
+ System.out.printf("\tobject count: %d\n",
meter.countChildren(measureThis));
+
System.out.println("=============================================");
+ }
+
+ protected abstract Object runAndGetObjectToMeasure();
+ }
+
+
//---------------------------------------------STUBS-----------------------------------------------------------
+ public static MacPoolManagerOriginal createMacPoolManagerOriginalSpy(int
maxMacsInPool, boolean createMockitoSpy) {
+
+ if (createMockitoSpy) {
+ final MacPoolManagerOriginal spy = Mockito.spy(new
MacPoolManagerOriginal(maxMacsInPool, RANGES_STRING, ALLOW_DUPLICATES));
+
Mockito.doReturn(Collections.emptyList()).when(spy).getVmNicInterfacesFromDB();
+ Mockito.doNothing().when(spy).logMacPoolEmpty();
+
+ return spy;
+ } else {
+ return new MacPoolManagerOriginal(maxMacsInPool, RANGES_STRING,
ALLOW_DUPLICATES) {
+ @Override
+ List<VmNic> getVmNicInterfacesFromDB() {
+ return Collections.emptyList();
+ }
+
+ @Override
+ void logMacPoolEmpty() {
+ //do no nothing.
+ }
+ };
+ }
+
+ }
+
+ public static MacPoolManagerRanges createMacPoolManagerRangesSpy(int
maxMacsInPool, boolean createMockitoSpy) {
+ if (createMockitoSpy) {
+ final MacPoolManagerRanges spy = Mockito.spy(new
MacPoolManagerRanges(maxMacsInPool, RANGES_STRING, ALLOW_DUPLICATES));
+
Mockito.doReturn(Collections.emptyList()).when(spy).getVmNicInterfacesFromDB();
+ Mockito.doNothing().when(spy).logMacPoolEmpty();
+
+ return spy;
+ } else {
+
+ return new MacPoolManagerRanges(maxMacsInPool, RANGES_STRING,
ALLOW_DUPLICATES) {
+ @Override
+ List<VmNic> getVmNicInterfacesFromDB() {
+ return Collections.emptyList();
+ }
+
+ @Override
+ void logMacPoolEmpty() {
+ //do nothing.
+ }
+ };
+ }
+ }
+
+}
diff --git
a/backend/manager/modules/bll/src/test/java/org/ovirt/engine/core/bll/network/MacPoolManagerValidityTest.java
b/backend/manager/modules/bll/src/test/java/org/ovirt/engine/core/bll/network/MacPoolManagerValidityTest.java
new file mode 100644
index 0000000..32c6bfe
--- /dev/null
+++
b/backend/manager/modules/bll/src/test/java/org/ovirt/engine/core/bll/network/MacPoolManagerValidityTest.java
@@ -0,0 +1,363 @@
+package org.ovirt.engine.core.bll.network;
+
+import static org.hamcrest.CoreMatchers.instanceOf;
+import static org.hamcrest.CoreMatchers.is;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.fail;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Random;
+import java.util.Set;
+
+import org.junit.Ignore;
+import org.junit.Test;
+import org.mockito.Mockito;
+import org.ovirt.engine.core.common.errors.VdcBLLException;
+import org.ovirt.engine.core.common.errors.VdcBllErrors;
+import org.ovirt.engine.core.utils.MacAddressRangeUtils;
+
+@Ignore
+public class MacPoolManagerValidityTest {
+
+ //disable mockito when testing on large ranges -- it slows process down,
can even crash.
+ public static final boolean DO_VERIFICATIONS_USING_MOCKITO = false;
+ public static final int MAX_MACS_IN_POOL = 1000000;
+// public static final int MAX_MACS_IN_POOL = 100000;
+// public static final int MAX_MACS_IN_POOL = 1000;
+
+ private final long from =
MacAddressRangeUtils.macStringToLong(MacPoolManagerTestUtils.RANGES_FROM);
+ private final long to =
MacAddressRangeUtils.macStringToLong(MacPoolManagerTestUtils.RANGES_TO);
+ private MacPoolManagerOriginal original;
+ private MacPoolManagerRanges ranges;
+
+ public void init() {
+ original =
MacPoolManagerTestUtils.createMacPoolManagerOriginalSpy(MAX_MACS_IN_POOL,
DO_VERIFICATIONS_USING_MOCKITO);
+ original.initialize();
+
+ ranges =
MacPoolManagerTestUtils.createMacPoolManagerRangesSpy(MAX_MACS_IN_POOL,
DO_VERIFICATIONS_USING_MOCKITO);
+ ranges.initialize();
+ }
+
+ @Test
+ public void _testAllMacsAreInSameState() throws Exception {
+ init();
+ testAllMacsAreInSameStateImpl();
+ }
+
+ void testAllMacsAreInSameStateImpl() {
+ new ForAllMacsInRange(from, to, MAX_MACS_IN_POOL){
+
+ @Override
+ void action(long mac, String macStr) {
+ assertThat(original.isMacInUse(macStr),
is(ranges.isMacInUse(macStr)));
+ }
+ }.doAction();
+ }
+
+ @Test
+ public void _testSameNumberOfFreeMacs() throws Exception {
+ init();
+ testSameNumberOfFreeMacsImpl();
+ }
+
+ void testSameNumberOfFreeMacsImpl() {
+ assertThat(original.getAvailableMacsCount(),
is(ranges.getAvailableMacsCount()));
+ }
+
+
+ /**
+ * pick some MACs, add them, compare, remove subset of them and compare.
+ * Run insane number of time for bigger pleasure.
+ */
+ @Test
+ public void addRemoveComparison() throws Exception {
+ final int CONVINCEMENT_DEGREE = 1;
+ final int NUMBER_OF_MACS_TO_DROP = MAX_MACS_IN_POOL / 4;
+ final int NUMBER_OF_MACS_TO_ADD = MAX_MACS_IN_POOL / 2;
+
+ System.out.println("addRemoveComparison");
+ long rangesAddTimeSum = 0;
+ long originalAddTimeSum = 0;
+ long rangesRmTimeSum = 0;
+ long originalRmTimeSum = 0;
+
+ for(int i = 0; i < CONVINCEMENT_DEGREE; i++) {
+ init();
+
+ //get all macs present in pool
+ ArrayList<Long> addedMacs = macsToAdd(true, NUMBER_OF_MACS_TO_ADD);
+
+ originalAddTimeSum += addMacs(addedMacs, original);
+ rangesAddTimeSum += addMacs(addedMacs, ranges);
+
+ if (DO_VERIFICATIONS_USING_MOCKITO) {
+ Mockito.verify(original).logMacPoolEmpty();
+ Mockito.verify(ranges).logMacPoolEmpty();
+ }
+
+ testAllMacsAreInSameStateImpl();
+ testSameNumberOfFreeMacsImpl();
+
+ List<String> macsToDrop1 = macsToDrop(addedMacs,
NUMBER_OF_MACS_TO_DROP);
+ List<String> macsToDrop2 = macsToDrop(addedMacs,
NUMBER_OF_MACS_TO_DROP);
+
+ originalRmTimeSum += removeMacsUsingTwoMethods(macsToDrop1,
macsToDrop2, original);
+ rangesRmTimeSum += removeMacsUsingTwoMethods(macsToDrop1,
macsToDrop2, ranges);
+
+ testAllMacsAreInSameStateImpl();
+ testSameNumberOfFreeMacsImpl();
+ }
+
+ System.out.println("\toriginalAdd: "+(originalAddTimeSum /
CONVINCEMENT_DEGREE / MacPoolManagerTestUtils.NANOS_IN_MS)+"ms");
+ System.out.println("\toriginalRm: "+(originalRmTimeSum /
CONVINCEMENT_DEGREE / MacPoolManagerTestUtils.NANOS_IN_MS) + "ms");
+ System.out.println("\trangesAdd: "+(rangesAddTimeSum /
CONVINCEMENT_DEGREE / MacPoolManagerTestUtils.NANOS_IN_MS) + "ms");
+ System.out.println("\trangesRm: "+(rangesRmTimeSum /
CONVINCEMENT_DEGREE / MacPoolManagerTestUtils.NANOS_IN_MS) + "ms");
+ }
+
+ private long removeMacsUsingTwoMethods(List<String> macsToDrop1,
List<String> macsToDrop2, MacPoolManagerStrategy impl) {
+ long start = MacPoolManagerTestUtils.timeStamp();
+
+ for (String mac : macsToDrop1) {
+ impl.freeMac(mac);
+ }
+
+ impl.freeMacs(macsToDrop2);
+ long end = MacPoolManagerTestUtils.timeStamp();
+ return end - start;
+ }
+
+ long addMacs(ArrayList<Long> addedMacs, MacPoolManagerStrategy impl) {
+ long start = MacPoolManagerTestUtils.timeStamp();
+ for (Long mac : addedMacs) {
+ impl.addMac(MacAddressRangeUtils.macAddressIntToString(mac));
+ }
+ long end = MacPoolManagerTestUtils.timeStamp();
+ return end-start;
+ }
+
+ private List<String> macsToDrop(ArrayList<Long> addedMacs, int
numberOfMacsToDrop) {
+
+ Random rnd = new Random();
+ List<String> toDrop = new ArrayList<>(numberOfMacsToDrop);
+ for(int i = 0; i < numberOfMacsToDrop; i++) {
+ Long removedNumber =
addedMacs.remove(rnd.nextInt(addedMacs.size()));
+
toDrop.add(MacAddressRangeUtils.macAddressIntToString(removedNumber));
+ }
+
+ return toDrop;
+ }
+
+ ArrayList<Long> macsToAdd(boolean noUserMacs, int numberOfMacsToAdd) {
+ Random rnd = new Random();
+ Set<Long> rnds = new HashSet<>(numberOfMacsToAdd);
+ int upperBound = upperBound(noUserMacs);
+ while (rnds.size() != numberOfMacsToAdd) {
+ long mac = rnd.nextInt(upperBound) + from;
+ rnds.add(mac);
+ }
+ return new ArrayList<>(rnds);
+ }
+
+ private int upperBound(boolean noUserMacs) {
+ if (noUserMacs) {
+ return MAX_MACS_IN_POOL;
+ } else {
+ return (int) (to - from + 1);
+ }
+ }
+
+
+ @Test
+ public void fullFillTest() throws Exception {
+ System.out.println("fullFillTest");
+ init();
+ testAllMacsAreInSameStateImpl();
+ testSameNumberOfFreeMacsImpl();
+
+ long originalAddTime = addAllMacs(original, false);
+ long rangesAddTime = addAllMacs(ranges, false);
+
+ if (DO_VERIFICATIONS_USING_MOCKITO) {
+ Mockito.verify(original).logMacPoolEmpty();
+ Mockito.verify(ranges).logMacPoolEmpty();
+ }
+
+ System.out.println("\toriginalAdd: " + (originalAddTime /
MacPoolManagerTestUtils.NANOS_IN_MS) + "ms");
+ System.out.println("\trangesAdd: " + (rangesAddTime /
MacPoolManagerTestUtils.NANOS_IN_MS) + "ms");
+
+ testAllMacsAreInSameStateImpl();
+ testSameNumberOfFreeMacsImpl();
+
+ assertThat(original.getAvailableMacsCount(), is(0));
+ assertThat(ranges.getAvailableMacsCount(), is(0));
+
+ unableToAllocateNewMac(original);
+ unableToAllocateNewMac(ranges);
+
+ allMacUsed(original);
+ allMacUsed(ranges);
+ }
+
+ @Test
+ public void fullFillTest2() throws Exception {
+ System.out.println("fullFillTest2");
+ init();
+ testAllMacsAreInSameStateImpl();
+ testSameNumberOfFreeMacsImpl();
+
+ long originalAddTime = addAllMacs(original, true);
+ long rangesAddTime = addAllMacs(ranges, true);
+
+ System.out.println("\toriginalAdd: " + (originalAddTime /
MacPoolManagerTestUtils.NANOS_IN_MS) + "ms");
+ System.out.println("\trangesAdd: " + (rangesAddTime /
MacPoolManagerTestUtils.NANOS_IN_MS) + "ms");
+
+ testAllMacsAreInSameStateImpl();
+ testSameNumberOfFreeMacsImpl();
+
+ assertThat(original.getAvailableMacsCount(), is(0));
+ assertThat(ranges.getAvailableMacsCount(), is(0));
+
+ unableToAllocateNewMac(original);
+ unableToAllocateNewMac(ranges);
+
+ allMacUsed(original);
+ allMacUsed(ranges);
+ }
+
+ @Test
+ public void allocatingOnTheEndOfArrayPerformance() throws Exception {
+ System.out.println("allocatingOnTheEndOfArrayPerformance");
+ init();
+
+ addAllMacs(original, false);
+ addAllMacs(ranges, false);
+
+ if (DO_VERIFICATIONS_USING_MOCKITO) {
+ Mockito.verify(original).logMacPoolEmpty();
+ Mockito.verify(ranges).logMacPoolEmpty();
+ }
+
+ assertThat(original.getAvailableMacsCount(), is(0));
+ assertThat(ranges.getAvailableMacsCount(), is(0));
+
+ LinkedList<String> macs = new LinkedList<>();
+ for(int i = 1; i < 1000; i++) {
+ final int maxMacsInPool = MAX_MACS_IN_POOL;
+ long mac = from + maxMacsInPool - i;
+ if (!MacAddressRangeUtils.macHasMultiCastBitSet(mac)) {
+ macs.add(MacAddressRangeUtils.macAddressIntToString(mac));
+ }
+ }
+
+ int repetitions = 50;
+ long originalAddTime = 0;
+ long rangesAddTime = 0;
+ long originalRmTime = 0;
+ long rangesRmTime = 0;
+ int numberOfMacsToAllocate = macs.size();
+
+ for(int i = 0; i < repetitions; i++) {
+ originalRmTime += freeTheseMacs(macs, original);
+ rangesRmTime += freeTheseMacs(macs, ranges);
+
+ assertThat(original.getAvailableMacsCount(),
is(numberOfMacsToAllocate));
+ assertThat(ranges.getAvailableMacsCount(),
is(numberOfMacsToAllocate));
+
+ originalAddTime += allocateNMacs(numberOfMacsToAllocate, original);
+ rangesAddTime += allocateNMacs(numberOfMacsToAllocate, ranges);
+
+ final int wantedNumberOfInvocations = (i + 1) + 1; //one time
per each repetition + once before this cycle.
+ if (DO_VERIFICATIONS_USING_MOCKITO) {
+ Mockito.verify(original,
Mockito.times(wantedNumberOfInvocations)).logMacPoolEmpty();
+ Mockito.verify(ranges,
Mockito.times(wantedNumberOfInvocations)).logMacPoolEmpty();
+ }
+
+ assertThat(original.getAvailableMacsCount(), is(0));
+ assertThat(ranges.getAvailableMacsCount(), is(0));
+ }
+
+ System.out.println("\toriginalAdd: " + (originalAddTime / repetitions
/ MacPoolManagerTestUtils.NANOS_IN_MS) + "ms");
+ System.out.println("\toriginalRm: " + (originalRmTime / repetitions/
MacPoolManagerTestUtils.NANOS_IN_MS) + "ms");
+ System.out.println("\trangesAdd: " + (rangesAddTime / repetitions/
MacPoolManagerTestUtils.NANOS_IN_MS) + "ms");
+ System.out.println("\trangesRm: " + (rangesRmTime / repetitions/
MacPoolManagerTestUtils.NANOS_IN_MS) + "ms");
+ }
+
+ private long allocateNMacs(int numberOfMacsToAllocate,
MacPoolManagerStrategy strategy) {
+ long start = MacPoolManagerTestUtils.timeStamp();
+ strategy.allocateMacAddresses(numberOfMacsToAllocate);
+ long end = MacPoolManagerTestUtils.timeStamp();
+ return (end - start);
+ }
+
+ long freeTheseMacs(LinkedList<String> macs, MacPoolManagerStrategy
strategy) {
+ long start = MacPoolManagerTestUtils.timeStamp();
+ strategy.freeMacs(macs);
+ long end = MacPoolManagerTestUtils.timeStamp();
+ return (end - start);
+ }
+
+ private void allMacUsed(final MacPoolManagerStrategy strategy) {
+ new ForAllMacsInRange(from, to, MAX_MACS_IN_POOL) {
+ @Override
+ void action(long mac, String macStr) {
+ assertThat("mac '" + mac + "' should be used but it's
not(index of mac=" + (mac - from) + ")", strategy.isMacInUse(macStr), is(true));
+ }
+ }.doAction();
+ }
+
+ void unableToAllocateNewMac(MacPoolManagerStrategy strategy) {
+ try {
+ strategy.allocateNewMac();
+ fail("expected " + VdcBLLException.class.getSimpleName() + "
exception during 'allocateNewMac on '" + strategy.getClass().getSimpleName());
+ } catch (Exception e) {
+ assertThat(e, instanceOf(VdcBLLException.class));
+ assertThat(((VdcBLLException)e).getErrorCode(),
is(VdcBllErrors.MAC_POOL_NO_MACS_LEFT));
+ }
+ }
+
+ long addAllMacs(final MacPoolManagerStrategy strategy, final boolean
usingAllocate) {
+ long start = MacPoolManagerTestUtils.timeStamp();
+
+ new ForAllMacsInRange(from, to, MAX_MACS_IN_POOL){
+ @Override
+ void action(long mac, String macStr) {
+ if (!usingAllocate) {
+ strategy.addMac(macStr);
+ } else {
+ strategy.allocateNewMac();
+ }
+ }
+ }.doAction();
+
+ long end = MacPoolManagerTestUtils.timeStamp();
+ return end - start;
+ }
+
+ private static abstract class ForAllMacsInRange {
+ private final long from;
+ private final long to;
+ private final int maxMacsInPool;
+
+ protected ForAllMacsInRange(long from, long to, int maxMacsInPool) {
+ this.from = from;
+ this.to = to;
+ this.maxMacsInPool = maxMacsInPool;
+ }
+
+ void doAction() {
+ for (long mac = from, count = 0; mac < to && count <
maxMacsInPool; mac++) {
+ if (!MacAddressRangeUtils.macHasMultiCastBitSet(mac)) {
+ count++;
+ }
+ String macStr =
MacAddressRangeUtils.macAddressIntToString(mac);
+
+ action(mac, macStr);
+ }
+ }
+
+ abstract void action(long mac, String macStr);
+ }
+}
diff --git
a/backend/manager/modules/bll/src/test/java/org/ovirt/engine/core/bll/network/VmInterfaceManagerTest.java
b/backend/manager/modules/bll/src/test/java/org/ovirt/engine/core/bll/network/VmInterfaceManagerTest.java
index 1afa5f0..f70bf11 100644
---
a/backend/manager/modules/bll/src/test/java/org/ovirt/engine/core/bll/network/VmInterfaceManagerTest.java
+++
b/backend/manager/modules/bll/src/test/java/org/ovirt/engine/core/bll/network/VmInterfaceManagerTest.java
@@ -49,7 +49,11 @@
@ClassRule
public static MockConfigRule mcr = new MockConfigRule(
- mockConfig(ConfigValues.HotPlugEnabled, VERSION_3_2.getValue(),
true));
+ mockConfig(ConfigValues.HotPlugEnabled, VERSION_3_2.getValue(),
true),
+ mockConfig(ConfigValues.MacPoolRanges,
"00:1a:4a:15:c0:00-00:1a:4a:15:c0:ff"),
+ mockConfig(ConfigValues.MaxMacsCountInPool, 100000),
+ mockConfig(ConfigValues.AllowDuplicateMacAddresses, false)
+ );
@Mock
private MacPoolManager macPoolManager;
--
To view, visit http://gerrit.ovirt.org/26405
To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I69a5c1b3b43966e49fa6039597c06966ce514618
Gerrit-PatchSet: 1
Gerrit-Project: ovirt-engine
Gerrit-Branch: master
Gerrit-Owner: Martin Mucha <mmu...@redhat.com>
_______________________________________________
Engine-patches mailing list
Engine-patches@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-patches
